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Memorandum  for  OER  Workshop  Participants/Attendees 
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Staff  Report,  since  attendees  were  present  for  briefings 
(Appendix  2),  and  Interim  Report  of  Panel  Proposals  (Appen¬ 
dices  3-7)  have  been  distributed  previously. 

The  preceding  Appendices  are  included  in  Staff  Report 
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USAF  OFFICER  EVALUATION  SYSTEM: 

REVIEW  AND  RESEARCH  RECOMMENDATIONS 

PURPOSE 

The  USAF  Deputy  Chief  of  Staff  for  Personnel  has  asked  that  a 
new  Officer  Evaluation  System  he  developed.  This  report  outlines  al¬ 
ternatives  and  development  approaches  created  by  the  Air  Force  Human 
Resources  Laboratory. 

BACKGROUND 

There  has  been  extensive  operational  experience  with  evaluation 
systems  in  all  branches  of  the  Armed  Forces,  other  government  agencies, 
and  industry.  Many  approaches  have  been  employed  and  in  some  cases  a 
given  organization  has  used  more  than  one  methodology.  A  modest  -  but 
continuing  -  analysis  of  data  generated  by  the  Air  Force  system  has  been 
under  way  for  several  years.  Similar  data  have  been  accrued  in  other  or¬ 
ganizations  and  a  reasonable  body  of  expertise  on  evaluation  methodologies 
has  developed  in  several  universities. 

Evaluation  systems  may  themselves  range  from  very  simple  to  quite 
complex.  But  within  the  Air  Force,  because  of  the  many  roles  for  which 
evaluations  are  used  and  the  diverse  circumstances  under  which  ratings  are 
rendered,  impact  of  the  system  used  is  considerable  -  on  the  individual 
as  well  as  on  the  Air  Force  and  the  country. 

Today's  Officer  Effectiveness  Reports  (OER)  influence  promotions, 
assignments,  career  and  performance  counseling,  and  fill  several  supple¬ 
mental  functions.  Adequateness  of  the  existing  system  varies  between 
the  functions  for  which  used,  but  the  dominant  visible  defect  is  inflation 


a  worsening  tendency  for  raters  to  rate  subordinates  at  the  highest  pos¬ 
sible  level.  This  makes  discrimination  between  officers,  especially  for 
promotion,  progressively  more  difficult. 

There  is  considerable  historical  evidence  to  indicate  that  the  risk 
of  failure  is  very  high  for  a  new  evaluation  methodology.  Systems  tried 
within  the  Air  Force  in  the  past  which  forced  discrimination,  or  mechan¬ 
ically  precluded  inflation,  proved  so  unpopular  that  they  were  discontinued. 
Various  efforts  to  avoid  inflation  in  systems  which  do  not  mechanically 
prevent  the  problem  have  uniformly  failed. 

AFHRL  APPROACH 


To  increase  the  odds  of  developing  a  discriminating  system  which 
could  be  accepted,  AFHRL  collected  the  best  experience  information  now 
available,  presented  the  data  and  the  existing  Air  Force  system  to  a 
Workshop  composed  of  national  experts  drawn  from  industry,  universities, 
government  laboratories,  and  the  Armed  Forces;  and  had  five  panels  composed 
of  these  experts  provide  recommendations  for  system  changes  in  development 
approaches.-*-  Participants  are  listed  in  Appendix  1. 

Documentation  on  present  Army,  Navy,  Marine  Corps,  Coast  Guard,  Air 

Force  (Military  and  Civilian),  General  Motors,  J.C.  Penney  Company,  IBM, 

Royal  Air  Force,  and  Royal  Australian  Air  Force  Systems  is  given  in  Appendix  2. 

Panel  recommended  systems  are  presented  in  Appendix  3  through  7.  Ap¬ 
pendix  8  is  a  sixth  approach  developed  independently  by  Lt  Colonel  Robert 
A.  Dineen  of  AFHRL. 

^Liberal  support  to  this  project  was  provided  by  the  Office  of  Scientific 
Research  which  managed  the  contractual  part.  University  of  Maryland,  Purdue 
University,  Kent  State  University,  Arizona  State  University,  Medical  College 
of  Georgia,  General  Motors  Corporation,  International  Business  Machines,  Inc., 
American  Institute  for  Research,  Personnel  Decisions,  Inc  (University  of 
Minnesota),  Fieldcrest  Mills,  Inc.,  Royal  Air  Force,  Royal  Australian  Air 
Force,  Office  of  Naval  Research,  Coast  Guard,  Marine  Corps,  Army  Staff, 

Navy  Staff,  and  Air  Staff. 
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COMPARISON  OF  WORKSHOP  PANEL  PROPOSALS 


The  USAF  Officer  Evaluation  System  Workshop  was  comprised  of  five  work¬ 
ing  Panels  of  personnel  management  specialists  from  industry,  academic  insti¬ 
tutions,  the  military  services  and  various  government  agencies.  The  Panels 
were  known  as  the  Crimson,  Red,  Orange,  Maroon  and  Scarlet  Panels  -  Appendices 
3-7  lists  the  membership  of  each  of  the  Panels.  The  Workshop  Panels  were 
assigned  four  major  tasks,  namely,  (l)  produce  at  least  one  evaluation  sys¬ 
tem  proposal  with  recommendations  for  follow-on  research;  (2)  prepare  pre¬ 
sentations  of  proposals  for  review  and  critique  by  the  Workshop;  (3)  prepare 
a  report  of  proposals  and  recommendations;  (^)  academic/industrial  consul¬ 
tants  critique  the  Panel  Reports  subsequent  to  completion  of  the  Workshop. 

The  five  Workshop  Panels  deliberating  independently,  developed  eight 
distinct  recommendations  which  were  called  the  Crimson  A  and  Crimson  B 
Proposal;  the  Red  A,  Red  B  and  Red  C  Proposal;  the  Orange  Proposal,  the 
Maroon  Proposal;  and  the  Scarlet  Proposal.  The  Panel  Reports  are  presented 
in  their  entirety  in  Appendices  3-7*  The  present  section  summarizes  areas  of 
consensus  in  the  Panel  approaches  and  briefly  considers  a  few  of  the  dis¬ 
tinctive  and  unique  features  of  each  of  the  proposals. 

Promotion  Potential  Evaluations  and  Feedback 

The  Red  A  and  B,  Scarlet,  Orange,  and  Crimson  A  proposals  all  provide 
for  a  separate  promotion  potential  evaluation  document,  which  to  varying  de¬ 
grees  would  be  maintained  as  "confidential."  The  Red  A  proposal  recommends 
that  the  evaluation  document  be  reviewed  by  the  indorsing  official.  The 
promotion  potential  document  would  then  be  directly  forwarded  to  the  Selection 
Secretariat's  file.  Rating  distributions  by  grade  and  command  would  be  pro¬ 
vided  to  all  raters  and  ratees.  Ratee  feedback  would  also  include  a  comparison 


3 


of  the  individual's  relative  standing  with  his  contemporaries  and  an  esti¬ 
mate  of  promotion  probability  (in  percentage  terms)  based  on  regression 
analysis.  The  Red  B  Proposal  is  similar  to  the  Red  A  Proposal,  except  that 
it  allows  ratees  to  review  appraisals  after  promotion  action.  The  Scarlet 
Proposal  suggests  a  separate,  or  detachable  form  (from  the  performance  evalu¬ 
ation)  that  includes  a  promotion  potential  score  which  would  be  confidential. 
The  Proposal  of  the  Orange  Panel  includes  a  separate  confidential  promotion  po¬ 
tential  rating  and  provides  for  ratee  feedback  at  critical  stages.  For  career 
officers,  critical  stage  reviews  would  be  conducted  upon  entering  the  secon¬ 
dary  and  primary  zones  of  eligibility  for  promotion  to  major,  lieutenant 
colonel  and  colonel.  A  second  independent  promotion  potential  evaluation 
would  also  be  accomplished  by  the  second  echelon  supervisor.  In  addition, 
provision  is  made  in  the  Orange  Proposal  for  a  base  level  review  board  for 
quality  control  purposes. 

The  Crimson  A  plan  provides  for  a  confidential  promotion  potential 
rating  with  a  one-time  feedback  of  rating  distribution  to  raters  and  a 
"no  disclosure  policy"  thereafter.  The  second  and  third  echelon  super¬ 
visors  may  also  accomplish  promotion  potential  evaluations  on  the  ratee  under 
the  Crimson  A  system,  but  such  ratings  would  be  independent  and  there  would 
be  no  indorsement  or  review  procedure  for  any  of  the  promotion  potential 
ratings.  Ratees  would  be  assigned  a  0  to  99  score  in  promotion  potential  with 
no  rater  being  permitted  to  rate  any  two  of  his  ratees  with  the  same  score 
(no  ties ) . 

The  Red  C,  Maroon  and  Crimson  B  proffered  evaluation  systems  are  all 


k 


"open  systems,"  i.e.,  ratees  would  have  access  to  their  promotion  potential 
ratings.  The  Maroon  and  Crimson  B  Proposals  advocate  administrative  con¬ 
trol  (of  inflation)  procedures.  Under  the  Maroon  system  called  PAT  (Point 
Allocation  Technique),  a  minimum  of  three  officers  would  be  evaluated  simul¬ 
taneously.  If  a  particular  supervisor  does  not  have  three  ratees,  he  would 
pool  his  ratees  with  those  of  his  supervisor,  and  so  forth,  until  a  minimum 
ratee  group  is  established.  If  necessary,  the  two  or  more  supervisors  form 
an  evaluation  panel  for  the  purpose  of  allocating  points.  Each  rater  has 
100  points  per  ratee  for  each  factor  considered  and  these  are  distributed 
among  the  pool  to  establish  an  ordering  of  ratees.  A  standardization  pro¬ 
cedure  is  then  used  to  evaluate  the  USAF-wide  population  for  each  grade. 

The  Crimson  B  system  advocates  that  the  present  OER  system  be  retained,  but 
that  each  MAJCOM  would  be  given  a  mandatory  quota  for  specific  rating  levels 
for  distribution  throughout  the  command. 

Single  Versus  Multipurpose  Forms 

A  number  of  proposals  (Orange,  Scarlet,  Crimson  A,  Red  A  and  Red  B) 
recommended  that  different  forms  be  used  for  different  purposes  rather  than 
using  a  multipurpose  form  as  is  presently  the  case.  In  general,  the  recom¬ 
mendation  would  be  to  provide  one  form  for  job  description,  another  form  for 
job  performance  evaluation,  another  form  for  promotion  potential  and  another 
form  for  job  and/or  career  counselling.  The  Red  C  Proposal  provides  for 
separate  self-appraisal  and  a  rater  history  forms,  but  includes  a  multipurpose 
form  for  the  functions  of  performance  evaluation,  promotion  potential  evalu¬ 
ation,  and  counselling.  The  Maroon,  and  Crimson  B  system  proposals  are  based 
on  multipurpose  forms  similar  to  the  present  AF  Forms  77  and  707» 
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Simultaneous  Submission  of  Ratings 


With  regard  to  the  timing  of  submittal  of  promotion  potential  ratings 
for  career  officers,  all  Panel  Proposals,  with  the  exception  of  the  Red 
and  Crimson  B,  were  in  agreement  that  ratings  should  normally  be  submitted 
annually  and  simultaneously  by  grade.  For  example,  all  annual  promotion 
potential  evaluations  would  be  due  March  1  for  captains,  July  1  for  majors, 
etc.  This  change  would  provide  raters  and  all  levels  of  review  a  means 
for  simultaneously  and  directly  comparing  the  promotion  potential  reports 
of  all  officers  of  the  same  grade. 

Job  Description 

Three  of  the  Panels  (Crimson,  Orange  and  Red)  addressed  the  job  descrip¬ 
tion  aspect  of  the  evaluation  system.  All  were  in  agreement  that  the  ratee 
should  provide  an  input  into  preparation  of  the  job  description.  The  Crimson  A 
Proposal  provides  for  preparation  and  updating  of  the  job  description  by  the 
supervisor  through  discussion  with  the  ratee.  Under  this  system,  the  super¬ 
visor  is  to  assign  quantitative  scores  using  benchmark  scales  for  each  position. 
The  Orange  Panel  Proposal  provides  for  all  ratees  to  prepare  the  job  description 
and  quantitatively  evaluate  the  position  as  to  difficulty.  The  job  description 
would  then  be  reviewed  and  approved  or  adjusted,  if  necessary,  by  the  rater. 

The  Red  A  Proposal  recommends  joint  preparation  and  agreement  upon  specific 
duty  description  at  the  start  of  the  rating  period.  The  indorser  would  re¬ 
view  the  duty  description  and  it  would  then  become  available  to  Military 
Personnel  Center  Assignments.  In  addition  to  the  Red  A  requirements  for  a 
jointly  prepared  duty  description,  the  Red  B  Proposal  provides  that  a  formal 
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job- oriented  appraisal  be  included.  The  Scarlet  and  Maroon  panels  did  not 
consider  this  aspect  of  the  officer  evaluation  system. 

First-Tour  Officers 

The  Orange,  Maroon,  and  Red  Panels  consider  the  problem  of  an  appraisal 
system  for  first-tour  officers  as  distinct  from  that  of  career  officers. 
First-tour  officers  were  viewed  as  requiring  a  separate  appraisal  system 
which  provides  for  career  counseling,  guidance  and  development,  job  per¬ 
formance  and  augmentation  evaluation  and  assignments  evaluation.  It  was 
generally  recommended  that  the  first-tour  officer  evaluation  system  be  an 
open  system. 

Education  and  Training  Program 

The  need  for  a  continuing  education  and  training  program  directed  at 
educating  raters  and  ratees  on  the  evaluation  system  requirements  of  the  Air 
Force  was  considered  by  all  Panels.  All  Panels  agreed  that  an  education  and 
training  program  was  necessary  for  the  implementation  and  continued  proper 
operation  of  any  new  system.  Various  methods  of  instruction  including  Air 
Force-wide  classroom  and  home  study  courses,  training  films,  programmed  in¬ 
struction  booklets,  and  the  use  of  travelling  teams  of  instructors  were  recom¬ 
mended.  Inclusion  of  a  block  of  instruction  in  the  PME,  AFROTC,  OTS,  AFA 
programs  was  also  suggested. 

Pat  a  Aut  omat i on 

All  Panels  which  considered  the  topic  of  data  automation  of  the  evalu¬ 
ation  system  recommended  provision  for  optical  scanning  of  the  reports  and 
the  establishment  of  a  data  bank  as  a  basis  for  continuing  analysis  and  long¬ 
term  research. 
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SOLUTION  ALTERNATIVES 


General 

There  is  a  general  consensus  among  our  experts  that  multiple  purposes 
for  which  the  present  OER  is  used  cannot  test  he  satisfied  by  a  single  docu¬ 
ment.  Separate  job  description,  job  performance,  and  promotion  potential 
evaluation  systems  are  recommended.  Further,  instruments  required  for  grades 
2d  Lieutenant  through  First  -Tour  Captain  should  differ  from  those  used  for 

senior  Captain  through  Lt  Colonel.  Our  focus  is  on  the  latter  group. 

Job  Description 

The  job  description  is  a  basic  building  block  for  an  overall  evaluation 
system.  It  should  be  carefully  developed  and  adequately  used.  Production 
of  this  instrument  is  a  rater  responsibility  in  interaction  with  the  ratee. 
Unfortunately,  the  relationship  between  the  job  description  and  performance 
evaluation  has  not  been  satisfactorily  emphasized  in  the  operating  system. 
Guidance  and  standardization  of  how  this  tool  will  be  developed  has  not  been 
provided.  We  recommend  that  standardized  guidance  be  published  by  Hq  USAF  at 
an  early  date.  No  research  and  development  is  needed  to  facilitate  such  in¬ 
structions. 

Job  responsibility  and  job  requirement  considerations  are  implicit  to 
the  assignment  and  promotion  activities.  A  methodology  for  defining  and  rating 
job  requirement  factors  was  developed  by  the  Personnel  Division  in  196k. 
M/General  Harry  J.  Sands  and  his  USAF  Study  Group  on  Specialized  Officer  Career 
Development  recommended  in  1966  that  this  methodology  be  applied  USAF-wide. 

It  would  give  additional  relevant  discrimination.  We  recommend  that  application 
of  the  methodology  by  supervisors  to  job  descriptions  and  routine  reporting  of 
results  to  indorsers,  promotion  and  assignment  activities  be  required.  (Appen¬ 
dix  9  contains  the  methodology,  but  updating  is  needed.) 
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Performance  Evaluation 


Job  performance  is  highly  relevant  to  assignment,  but  not  completely 
correlated  with  promotion  potential.  An  officer  may  deserve  high  per¬ 
formance  marks,  yet  lack  promotion  potential.  Another  officer  may  be  work¬ 
ing  in  a  new  area  intentionally  given  for  career  broadening.  Short  range 
performance  can  be  expected  to  drop;  but  promotion  potential  may  be  enhanced. 

The  Workshop  concluded  that  separation  of  performance  and  promotion 
potential  ratings  is  not  only  needed,  but  that  performance  evaluation 
should  be  made  more  specific  to  the  job  and  individual.  The  rating  instru¬ 
ment  should  be  composed  of  multiple  factors  tailored  to  the  occupational 
area.  Raters  should  be  asked  to  mark  only  the  factors  relevant  to  the 

O 

specific  job.  The  performance  evaluation  would  be  openly  discussed  with 
the  subordinate,  then  forwarded  to  the  assignment  activity  and  to  the  in¬ 
dorser. 

To  initiate  this  performance  evaluation  system,  it  is  recommended 
that  career  field  forms  initially  be  created  for  the  ten  major  officer  career 
areas.  Let  raters  add  relevant  factors  as  the  system  operates  and  further 
break-out  and  augment  the  forms  from  these  field  generated  data. 

Promotion  Potential 

Needs  of  the  service,  equity  for  officers,  and  uncertainty  are  key 
components  of  promotion  potential  ratings.  USAF  interests  require  that  the 

p 

This  is  modeled  after  best  features  of  the  USAF  Civilian  Evaluation 
System. 
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evaluation  system  give  sufficient  discrimination  to  permit  selection  of 
the  officer  whose  future  performance  will  best  meet  service  needs.... But 
promotion  is  the  dominant  tangible  reward  -  and  a  key  motivator  -  for 
current  performance.  This  creates  pressures  for  inflation. 

Two  approaches  can  be  used  to  attack  inflation  in  today's  system, 
assuming  no  fundamental  changes.  One  approach  involves  making  ratings 
confidential.  This  removes  the  pressure  on  the  rater  to  inflate  in  order 
to  motivate  -  or  to  avoid  telling  the  ratee  where  he  stands.  The  second 
approach  involves  imposing  controls  to  force  discrimination  in  ratings. 

Since  basic  changes  which  remove  the  fundamental  assumptions  of  to¬ 
day's  system  can  be  recommended  only  after  realistic  field  testing  -  and 
since  such  field  testing  will  take  several  months  -  the  policy  maker  is 
limited  in  the  short-range  actions  that  he  can  take. 

Some  specific  suggestions  developed  by  the  Workshop,  which  may  give 
short-run  relief  include,  (l)  Quotas  (of  9s,  8s,  etc.)  for  major  com¬ 
mands;  (2)  Base  level  committee  review  of  ratings;  (3)  Indorser's  quotas; 

(b)  Confidential  ratings;  (5)  Eater  feedback  on  distribution  of  actual 
rating;  (6)  Training  and  education  of  raters;  (T)  Recorded  and  visible 
rater  histories. 

Recommendations  on  each  of  these  alternatives  are  provided  in  the  indi¬ 
vidual  Panel  Reports.  Further,  in  addition  to  recommending  two  systems  for 
field  test  (to  be  described  later),  we  offer  the  following  system  for  possible 
early  implementation  without  necessity  for  field  test.  The  system  does  not 

mechanically  insure  discrimination,  but  is  clear  and  uncomplicated. 

Early  Implementation  System 

The  underlying  theme  is  that  all  echelons  of  command  must  recognize  the 
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problem  of  people,  yet  at  the  same  time  accept  the  dictum  that  loyalty  is 
to  the  Air  Force  first,  people  second.  The  indorser  plays  a  critical  role 
insuring  loyalty  goes  "up"  to  the  Air  Force,  not  "down"  to  the  individual 
officer.  He  controls  personnel  appraisals,  inhibiting  rater  tendencies  to 
inflate  ratings.  Most  important  is  control  over  promotion  potential  apprai¬ 
sals.  If  an  indorser  feels  a  certain  rating  is  unjustified,  he  must  gain 
satisfaction  from  the  rater,  i.e.,  the  rater  is  held  responsible  for  each 
appraisal.  Indorsers  either  concur  in  the  appraisal,  or  execute  a  separate, 
independently  prepared  promotion  potential  appraisal. 

Two  other  appraisals  prepared  for  each  ratee  and  forwarded  to  the 
indorser  are  on  job  description  and  job  performance.  A  fourth  document, 
rater  history,  indicates  rank,  job  title,  and  promotion  potential  evaluations 
of  past  subordinates.  It  too  is  forwarded  to  indorsers.  All  four  documents 
are  weighted  relative  to  consistency  (or  inconsistency)  of  rater  appraisals, 
all  familiar  officers  of  same  rank,  requirements  of  the  Air  Force,  and  environ 
ment  in  which  the  ratee  was  working.  Promotion  potential,  job  descriptions, 
and  job  performance  appraisals  are  forwarded  to  the  Assignment  Branch  of  MPC. 
The  Selection  Branch  does  not  receive  job  performance  appraisals.  Appraisals 
schedules  are  in  accordance  with  current  regulations. 

The  promotion  potential  appraisal  is  confidentially  prepared  on  a  form 
similar  to  AF  Form  77  and  707  less,  specifically,  overall  evaluation,  job 
duties,  word  picture  and  promotion  potential  as  presently  scaled.  Promotion 
potential  should  be  scaled  in  the  following  manner:  Three  years  ahead  of 
contemporaries;  Two  years  ahead  of  contemporaries;  Top  50%  of  those  one  year 
ahead  of  contemporaries;  Lower  50%  of  those  one  year  ahead  of  contemporaries; 
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Top  third  of  those  with  contemporaries;  Middle  third  of  those  with  con¬ 
temporaries;  Lower  third  of  those  with  contemporaries;  One  year  behind  con¬ 
temporaries;  Do  not  promote. 

The  confidential  appraisal  is  forwarded  by  the  indorser  to  both  selec¬ 
tion  and  assignment  offices.  The  ratee  has  the  option  of  including  in 
his  selection  folder  relevant  information  pertaining  to  any  of  his  apprai¬ 
sals.  Such  information  goes  directly  to  the  selection  office,  by-passing 
raters  and  indorsers. 

The  ratee' s  "open  communication"  is  common  with  the  two  methods  recommended 
for  testing  in  the  field.  Two  other  common  features,  ratee  and  rater  feedback 
procedures,  are  discussed  in  the  context  of  these  two  methods. 

ALTERNATIVES  FOR  FIELD  TEST 

The  Air  Force  Human  Resources  Laboratory  is  recommending  that  two 
methods  for  insuring  discrimination  be  field  tested.  Each  approach  has 
certain  key  common  features. 

Rater  Feedback  -  both  approaches  give  the  rater  annual  feedback  on 
mean  and  distribution  of  recent  ratings.  They  provide  a  visible  standard  for  the 
rater  to  use  in  assessing  the  ratee. 

Ratee  Feedback  -  one  recommended  system  is  completely  open  with  instant 
ratee  feedback;  the  other  is  confidential  on  specific  ratings,  but  gives 
ratee  feedback  at  key  career  decision  points  on  how  the  individual  compares 
with  his  contemporaries  in  terms  of  promotion  probabilities. 

Annual  Calendar  Ratings  -  both  approaches  require  that  ratings  for  a 

* 

given  grade  be  as  of  a  given  date.  This  is  necessary  to  facilitate  com¬ 
parisons. 
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Open  Communication  -  both  approaches  should  be  complemented  by  a 
USAF  policy  of  open  communication  with  promotion  activities.  Present 
policy  on  communicating  with  permanent  boards  needs  be  extended  to  temporary 
boards. 

The  systems  being  recommended  for  test  involve  variations  of  rank 
ordering  within  a  ratee  group.  This  methodology  forces  discrimination, 
provided  ties  are  not  permitted.  While  such  a  system  insures  that  infla-c 
tion  is  eliminated,  it  has  off-setting  undesirable  features  which  require 
study  and  field-testing  with  respect  to  size  of  ratee  group.  Available  data 
indicate  that  the  number  of  officers  rated  by  most  first  level  supervisors 
is  very  small.  A  pool  of  at  least  three  appears  to  be  necessary  for  either 
system  to  work  well.  Four  primary  means  have  been  identified  for  expanding 
pool  size  when  required: 

.  Have  supervisor  rank  order  -  rate  not  only  his  subordinates,  but 
also  others  whom  he  knows  and  feels  competent  to  rate.  This  has  the  ad¬ 
vantage  of  expanding  the  ratee  pool,  but  it  carries  both  practical  and 
philosophical  weaknesses.  Adequate  identification  of  non-subordinates 
would  be  difficult.  Exposing  the  junior  officer  to  evaluation  by  a  senior 
who  might  be  on  the  opposite  side  in  critical  decision  issues  might  dampen 
the  vigor  with  which  alternatives  are  presented . Standards  for  rating  non¬ 

subordinates  would  probably  differ  among  raters.  Safeguards  would  probably 
be  necessary  to  prevent  use  of  non-subordinates  as  leverage  to  inflate 
rating  of  subordinates,  etc. 

.  Have  rater  rank  order  all  his  subordinates  without  regard  to  grade. 

This  alternative  also  increases  rating  pool  size,  but  implicitly  assumes 
that  all  grades  compete  for  the  same  higher  grade  level.  It  eliminates 
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the  sequential  ordering  by  grade  that  has  been  traditional. 

.  Have  rater  rank  order  ratee  in  population  of  people  whom  he  has  rated 
in  the  past.  This  alternative  probably  requires  that  ratee  population  in¬ 
clude  more  than  one  grade.  It  would  require  assumption  of  incremental  ad¬ 
ministrative  duties  by  a  central  activity.  A  definitive  approach  for  this 
alternative  has  been  developed  by  Lt  Colonel  Dineen  of  the  Personnel  Division. 
We  are  abstaining  from  recommending  a  field  test  at  this  time  because  com¬ 
plexity  of  the  system  is  such  that  user  communication  would  be  possible 
only  with  an  extensive  educational  program.  That  need,  plus  the  requirement 
of  a  limited  field  test  to  identify  and  correct  administrative  and  other 
problems,  make  us  categorize  the  approach  as  having  merit,  but  requiring 
longer  developmental  time  than  other  identified  alternatives. 

.  Merge  small  rater  pools,  with  second  level  supervisor  making  rank  order¬ 
ing  after  hearing  advice  by  first  level  supervisors.  While  this  approach  lacks 
optimum  rater  exposure  to  ratee,  it  appears  to  be  the  least  undesirable  of 
feasible  ways  identified  to  insure  that  each  ratee  is  compared  to  a  reason¬ 
able  number  of  his  peers  by  a  line  supervisor.  AFHRL  recommends  that  that 
two  variations  of  this  method  of  getting  a  satisfactory  ratee  pool  be  field- 
tested. 

Promotion  Potential  Ratings  (Fresca  Proposal,  Free  Scale  -  No  Ties) 

General  -  System  applies  to  senior  Captains,  Majors,  and  Lt  Colonels. 

All  officers  of  a  given  grade  are  rated  simultaneously,  with  exceptions  as 
specified  in  rationale  paragraph  below.  The  system  is  to  be  completely  auto¬ 
mated,  with  ratings  collected  on  forms  designed  for  optical  scanning. 

Specific  -  Definition  of  Raters  and  Ratee  Groups:  Ratings  are  to  be 
executed  by  all  officers  and  civilians  who  are  the  first  or  second  echelon 
supervisors  (current  rating  and  indorsing  officials)  of  one  or  more  individuals 


in  the  grade  under  consideration.  The  ratee  group  for  a  particular  rater  con¬ 
sists  of  all  officers  within  the  grade  under  consideration  for  whom  he  is  the 
first  or  second  echelon  supervisor.  If  a  ratee  group  consists  of  more  than  ten, 
but  fewer  than  twenty  members,  it  shall  be  sub-divided  into  two  ratee  groups  pf 
approximate  equal  size.  If  it  contains  more  than  20  but  fewer  than  30  members, 
it  shall  be  divided  into  three  groups  of  approximate  equal  size,  and  so  on. 

Rating  Procedure:  Members  of  each  ratee  group  shall  be  rated  on  an 
overall  promotion  potential  factor,  as  well  as  on  a  limited  number  of 
general  subfactors  related  to  potential.  Ratings  are  to  be  assigned  using 
a  centile  scale  running  from  00  to  99,  with  the  restriction  that  no  two 
members  of  a  ratee  group  can  be  assigned  exactly  the  same  score  value  on  a 
given  factor.  The  frame  of  reference  is  "all  officers  of  like  grade,"  Second 
echelon  supervisors  are  encouraged  to  gather  information  from  immediate  super¬ 
visors  before  executing  their  ratings. 

Disposition  of  Ratings:  Ratings  are  to  be  sent  to  a  central  agency, 
directly  by  the  rating  official.  They  are  to  remain  forever  confidential, 
except  for  use  by  assignment  officers,  promotion  boards,  and  research  personnel. 

Rater  Feedback:  Raters  are  provided  feedback  concerning  the  overall 
distribution  of  ratings  collected  during  the  first  rating  cycle.  No  infor¬ 
mation  about  commands  or  occupational  categories  shall  be  published.  Feed¬ 
back  after  the  first  cycle  may  or  may  not  be  provided,  depending  upon  data 
trends . 

Ratee  Feedback:  At  specified  career  points,  every  officer  is  provided 
information  reflecting  his  promotion  probabilities  as  compared  with  hi?  con¬ 
temporaries.  These  probabilities  are  computed  using  all  available  information, 
including  previous  promotion  potential  ratings;  however,  this  information 
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shall  he  presented  in  a  form  which  makes  it  impossible  for  him  to  infer  the 
promotion  potential  ratings  were  given  to  him  by  a  particular  rating  official. 

Rationale  and  Supplementary  Information  -  Simultaneous  Annual  Ratings 
of  all  Officers  in  Grade:  This  makes  possible  the  formation  of  ratee  groups 
which,  in  combination  with  the  restriction  of  no-ties  in  assigned  ratings, 
yields  data  from  which  man-to-man  rankings  can  be  derived.  If  also  makes 
officer  evaluation  a  scheduled  annual  event,  rather  than  a  chore  which  must 
be  performed  on  an  unscheduled  basis. 

Use  of  Centile  Scores:  Defines  an  initial  frame  of  reference  $nd 
gives  freedom  to  rater  for  expressing  his  convictions  concerning  the  potential 
of  his  subordinates.  Any  part  or  all  of  the  scale  may  be  used,  and  the  per¬ 
ceived  distances  between  ratees  can  be  indicated.  This  freedom  will  contribute 
to  user  acceptability. 

No-Ties  Feature:  Makes  possible  the  derivation  of  ranking  data.  Forces 
some  discrimination  to  be  made  at  the  local  level  by  those  individuals  who 


are  in  the  best  position  to  make  discriminations. 

Sub-Division  of  Large  Ratee  Groups:  It  is  recognized  that  the  no-tie 


feature  takes  some  freedom  away  from  the  rater.  For  example,  only  ten  in¬ 
dividuals  in  an  undivided  group  of  25  could  be  rated  above  the  90th  centile, 
even  in  an  "elite"  group  on  special  assignment.  It  is  evident  that  the  larger 
the  rating  group,,  the  less  freedom  the  rater  has  to  identify  ratees  in  a 
limited  range  of  the  scale.  Keeping  ratee  groups  at  size  ten  or  below  is 


a  compromise  between  giving  a  rater  complete  freedom,  and  forcing  him  tp 


make  discriminations. 


Sub factors  on  Promotion  Potential.  Provides  useful  information  to 
assignment  officers.  Makes  possible  the  derivation  of  differential  pro¬ 
motion  potential  composites  for  various  occupational  areas,  in  case  the 
Air  Force  should  ever  elect  to  move  away  from  the  concept  of  complete 
fungibility.  Permits  research  on  meaning  of  "overall  promotional  potential," 
as  presently  viewed  by  subclasses  of  rating  officers. 

Confidential  Ratings:  Takes  pressure  off  of  rater  to  inflate  evalu¬ 
ations  in  order  to  maximize  productivity  of  subordinates  having  "average" 
or  "below  average"  potential.  Potential  ratings  are  matters  of  judgment 
which  are  partially  based  upon  observation  of  characteristics  which  are 
not  modifiable  by  the  ratee.  Ratees  are  periodically  informed  concerning 
their  performance  level,  which  enters  into  decisions  about  promotion  potential 

Rater  Feedback:  It  is  hypothesized  that,  on  the  first  cycle,  some 
raters  will  follow  the  rules  and  assign  a  value  of  "50"  to  ratees  they 
perceive  as  being  average.  Other  raters  will  fear  that  inflation  will 
continue,  and  will  assign  a  value  higher  than  "50"  to  subordinates  they 
perceive  as  being  average  —  not  because  they  wish  to  be  dishonest  with 
the  system,  but  because  they  are  not  sure  what  the  system  is  going  to  be. 

The  feedback  of  the  overall  score  distribution  would  permit  everyone  to 
play  in  the  same  "ball  game."  Command  and  occupational  subgroup  means  should 
not  be  provided.  If  on  one  cycle  the  mean  ratings  for  a  particular  subgroup 
is  at  the  bottom  of  the  distribution  of  all  subgroup  means ,  there  is  a 
natural  tendency  for  members  of  that  subgroup  to  inflate  their  ratings. 

This  creates  a  "leap  frog"  effect,  since  some  group  mean  always  has  to  be 
on  bottom. 
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Ratee  Feedback:  Every  officer  deserves  to  know  how  he  stands  with 
respect  to  his  contemporaries  in  terms  of  promotion  probabilities.  life 
decisions  are  based  upon  such  information.  The  most  useful  feedback  would 
be  an  accurate  probability  statement,  which  should  weigh  together 'all  rele¬ 
vant  information. 

Out  of  Cycle  Ratings :  Since  the  system  permits  promotion  potential 
ratings  to  be  executed  on  a  single  individual,  such  ratings  could  be  required 
under  any  set  of  conditions  in  which  current  OERs  must  be  executed.  Alter- 
natives  are  possible,  such  as  requiring  an  officer  to  be  included  in  ratee 
groups  formed  at  the  annual  report  time,  even  though  the  offiqer  has  been 
reassigned  to  another  organization  during  the  report  period. 

First  and  Second  Echelon  Ratings:  During  data  reduction,  first  and 
second  echelon  supervisory  ratings  would  be  separated.  Raw  data  and 
composite  scores  derived  from  information  from  these  two  sources  would 
each  be  made  available  to  promotion  boards.  They  could  be  combined  if 
management  so  desires. 

Derivation  and  Applications  of  Rank  Information:  Since  no  ties  are 
permitted,  individuals  within  rating  groups  could  be  ranked  according  to 
score  level.  After  the  first  rating  cycle,  promotion  boards  conld  be  pro¬ 
vided  with  raw  rank  data,  e.g.,  "Ranked  second  in  a  group  of  six."  This 
information  could  be  used  in  conjunction  with  other  data  relating  to  job 
content  and  assignment  location  to  infer  quality.  After  several  rating  cycles, 
rank  data  could  be  merged  to  produce  a  partial  ordering  of  all  officers  being 
considered  in  a  promotion  cycle.  Note  that  this  ordering  would  not  only 
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consider  direct  comparisons,  but  inferred  transitive  relationships.  For 
example,  if  A  were  ranked  above  B  at  one  time  period,  and  B  were  ranked 
above  C  at  a  different  location  and  time  period,  it  can  be  inferred  that  A 
is  higher  than  C,  although  these  two  individuals  were  never  directly  compared. 
One  advantage  of  the  proposed  system  is  that  the  "no-ties"  feature  will  pro¬ 
duce  meaningful  information,  even  if  all  ratings  on  the  centile  score  are 
inflated. 

The  Point  Allocation  Technique ' (PAT) 

This  rating  system  is  designed  to  force  relative  rating  of  personnel, 
in  accord  with  the  competitive  nature  of  the  promotion  action.  The  comparison 
of  an  officer  with  other  officers  in  his  grade  is  best  done  at  the  lowest 
possible  level,  if  accurate  identification  of  relative  potential  is  to  be 
made . 

The  Point  Allocation  Technique  (PAT)  accepts  the  premise  that  selection 
for  successive  higher  grades  has  resulted  in  the  necessity  to  discriminate 
between  highly  qualified  individuals.  This  is  accomplished  through  the  rat¬ 
ing  device  of  providing  the  rating  officer  with  100  points  for  each  rating 
factor  for  each  officer  of  a  given  grade  whom  he  will  rate  in  a  given  cycle. 

If  there  is  no  detectible  difference  between  officers  in  their  demon¬ 
stration  of  a  given  factor,  they  receive  ratings  of  100.  As  differences  can 
be  determined,  points  are  taken  from  one  and  given  to  the  other ;  e.g, ,  values 
of  95  and  105  might  be  assigned  to  two  officers  who  demonstrated  a  small  dif¬ 
ference  in  potential;  90  and  110  would  be  more  striking;  75  and  125  would 
indicate  the  prescence  of  a  rare  and  uniquely  qualified  officer,  and  conversely, 
the  relatively  "slow  burners." 
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Promotion  potential,  being  a  comparative  quality,  should  be  evaluated 
in  a  group  of  not  less  than  3  officers.  In  the  case  of  a  rater  who  has  only 
one  or  two  ratees  of  a  given  grade,  it  is  necessary  that  he  go  up  one  eche¬ 
lon  to  his  supervisor,  in  order  that  his  ratee  can  be  pooled  against  other 
ratees  of  the  same  grade.  These  ratees  can  be  officers  of  the  appropriate 
grade  who  work  directly  for  the  supervisor,  or  for  whom  he  is  the  second 
echelon  supervisor.  When  this  occurs,  that  second  echelon  supervisor  and 
his  subordinate  raters,  serve  as  an  informal  selection  board  to  distribute 
points  among  the  ratees.  This  procedure  could  meet  the  desire  to  have  pro¬ 
motion  potential  evaluation  accomplished  at  the  lowest  possible  level,  within 
the  constraint  of  a  rating  group  of  not  less  than  3  officers.  It  has  the 
possibly  disabling  flaw  that  significant  percentages  of  rating  officers 
evaluate  only  one  or  two  ratees  who  are  in  grades  identical  to  their  own. 

A  procedure  for  handling  this  problem  must  be  developed  if  this  system  is  to 
be  implemented. 

It  is  recognized  that  this  forces  a  discrimination  between  recognized 
qualities  of  ratee  officers.  However,  it  is  pointed  out  that  such  dis¬ 
crimination  is  essential  to  identification  of  the  best  officers  fpr  promo¬ 
tion.  Historically,  promotion  boards  have  recognized  the  sure  promotee  and 
the  sure  passover  very  quickly;  the  difficult  decisions  occur  near  the  quota 
cutting  point.  This  system  offers  the  possibility  of  meaningful  discrimination 
across  a  rather  broad  spectrum. 

Values  distributed  in  this  way  would  be  helpful  in  the  assignment  process- 
under  current  standards  of  evaluations,  all  officers  are  highly  qualified  for 
all  positions. 
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The  rating  task  under  this  system  would  he  less  comfortable  than  the 
current  system.  It  places  the  rater  in  the  position  of  having  to  make 
discrimination  between  his  subordinates,  or  penalizing  all  of  them.  If  a 
spread  of  ratings  does  not  appear  in  the  ratee  group,  it  will  reduce  pro¬ 
motion  opportunity  for  every  member  of  that  group. 

In  order  to  stabilize  the  system  around  a  set  of  meaningful  values , 
and  to  provide  for  foreseen  management  contingencies,  the  following  rules 
for  evaluation  are  proposed: 

.  Each  rater  is  allocated  100  points  per  ratee  for  each  evaluation 
factor  which  he  is  to  distribute  among  his  ratees.  There  will  be  a  mini¬ 
mum  of  three  ratees  with  an  allocation  of  300  points. 

.  If  a  definitely  unsatisfactory  officer  is  in  the  group,  he  may  be 
assigned  a  value  of  TO  or  less;  if  this  occurs,  the  remainder  of  the  100 
points  allocated  drops  out  of  the  pool.  Here  is  an  example  of  ratings 
for  three  officers,  one  of  whom  is  unsatisfactory: 

65  +  110  +  90  =  265 

.  A  rating  higher  than  130  may  not  be  given.  This  rule  is  inserted 
to  prevent  grotesque  inflation  in  a  rating  of  group  of  12  to  15  officers. 

It  would  be  possible,  by  setting  the  majority  of  the  group  just  below  100 
to  produce  an  inordinately  high  value  for  a  single  ratee.  Applied  within 
the  limits  stated,  and  insuring  that  the  total  number  of  points  allocated 
is  equal  to  100  times  the  number  of  officers  (with  an  adjustment  for  points 
subtracted  because  of  ratings  equal  to  or  less  than  TO),  an  equitable  dis¬ 
tribution  of  ratings  will  appear. 

.  Every  rating  must  be  discussed  with  the  ratee.  This  rule  is  inserted 

to  provide  for  equity  in  rating — officers  have  a  recognition  of  the  comparative 

abilities  of  their  colleagues,  and  equitable  discrimination  will  not  be 
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offensive.  Knowing  that  each  of  his  ratees  must  he  told  the  rating  given 
will  do  a  great  deal  to  keep  a  rating  officer  objective,  and  supplying  data 
of  maximum  value  to  the  Air  Force. 

.  The  next  higher  echelon  of  supervisor  will  indorse  ratings  prepared 
by  his  subordinates.  Although  he  may  discuss  and  change  ratings  given, 
he  should  not  do  so  for  officers  with  whom  he  has  insufficient  face-to-face 
contact  to  be  certain  of  his  own  knowledge  of  the  relative  potentials.  A 
primary  function  of  the  indorser  is  to  insure  that  his  subordinates  have  in¬ 
troduced  reasonable  variation  into  their  ratings,  and  that  Air  Force  objec¬ 
tives  of  distribution  of  potential  to  permit  fair  promotion  opportunity 
throughout  the  Air  Force  are  met.  Of  course,  the  indorser  is  bound  by  the 
same  rules  as  the  rater. 

.  It  is  essential  that  the  rating  officer  understand  his  responsibility 
to  his  ratees.  If  he  assigns  the  standard  value  of  100  to  all  of  his  officers 
he  will  minimize  the  chances  of  their  promotion,  because  promotion  quotas 
above  about  U0  to  60$  of  the  population  are  rather  rare — mean  of  the  Air  Force 
wide  group  will  be  very  slightly  below  100  because  of  the  presence  of  men 
ranked  70  or  below  who  are  not  counter-balanced  by  a  "high"  ratee.  The 
circumstance  which  will  occur  is  shown  in  the  figure  below,  which  shows  the 
centil  equivalent  of  each  point  value,  as  well  as  a  comparison  of  two  raters 
and  an  Air  Force-wide  population.  Rater  A  has  given  scores  of  100  to  a 
majority  of  his  .ratees,  and  Rater  B  has  provided  greater  discrimination  be¬ 
tween  his  ratees. 
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used  in  the  AF  Form  TOT,  Field  Grade  Officer  Effectiveness  Report.  There  are 
certain  recommendations  for  changes  in  factors  to  he  evaluated,  hut  the  sig¬ 
nificant  difference  lies  in  the  conversion  to  a  relative  scale,  rather  than 
the  current  scale  with  its  attributes  of  absolute  assessment.  Various 
sections  of  the  proposed  system  will  he  addressed  below. 

The  Narrative  section  of  the  form  is  used  essentially  as  in  the  current 
system,  hut  emphasizes  a  requirement  of  a  description  of  tangible  results 
in  terms  of  the  Command  mission.  Discourage  adjectival  description  of  quality 
of  performance. 

Evaluation  Factors  or  Elements:  These  are  almost  exactly  as  in  current 
forms  and  are  :  (l)  Job  Capability,  appearing  on  present  AF  Form  TOT  is  de¬ 
leted,  to  be  replaced  by  a  different  factor,  later  in  the  scale;  (2)  Planning 
Ability,  unchanged  in  definition;  (3)  Executive  Management,  unchanged  in 
definition;  (4)  Leadership,  unchanged  in  definition;  (5)  Executive  Judgment, 
unchanged  in  definition;  and  (6)  Nurturing  Professional  Development  of  Sub¬ 
ordinates.  This  is  defined  in  terms  of  overt  activity  on  the  part  of  the 
ratee  to  improve  the  professional  competence  of  his  subordinates  and  to 
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contribute  to  their  maximum  growth  for  Air  Force  service.  It  is  changed- 
from  "Human  Relations"  on  the  present  AF  Form  707;  (7)  Writing  Ability,  de¬ 
fined  as  level  of  skill  in  preparation  of  written  reports,  correspondence, 
technical  material,  or  office  memoranda;  (8)  Speaking  Ability,  defined  as  skill 
in  the  communication  of  ideas  in  either  formal  or  informal  situations;  (9) 

Mission  Contribution.  This  factor  is  defined  in  terms  of  the  extent  to  which 
the  officer  has  forwarded  his  unit  mission  during  the  reporting  period. 

In  this  context,  it  has  some  relevance  to  the  "Job  Capability"  factor  deleted 
from  the  AF  Form  707 >  but  goes  beyond  this  to  assess  the  extent  to  which  the 
officer  is  oriented  to  accomplishing  the  mission  of  his  unit,  as  opposed  to 
the  less  complex  concept  of  display  of  simple  job  skills;  (10)  Military 
Qualities,  defined  as  on  AF  Form  707;  (ll)  Overall  Evaluation,  defined  as 
on  AF  Form  707. 

How  To  Test 

There  would  be  unacceptable  risk  of  failure  in  attempting  a  USAF-wide 
operational  application  of  any  basic  changes  to  the  existing  system  without 
prior  successful  testing.  Yet,  realistic,  meaningful  testing  in  a  non-operational 
environment  is  not  possible.  Simulation  does  not  give  the  personal,  psycho¬ 
logical  pressures  that  exist  in  the  real  world  and  conclusions  based  on  "dry 
runs"  not  expected  to  influence  promotion  decisions  could  be  misleading,  even 
dangerous . 

One  suggested  approach  that  we  do  offer  for  consideration  consists  of 
a  limited  but  real  application  of  the  methodologies  in  two  separate  organizations 
(commands).  Realism  would  be  provided  by  prior  announcement  that  results  of 
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evaluations  made  in  the  system  being  tested  would  be  used  to  distribute 
that  command's  fair  share  of  USAF  promotion  quotas  for  the  year.  A  separate 
Promotion  Secretariat  Panel  would  be  tasked  with  making  such  quota  allocations 
within  each  grade  using  evaluations  from  the  new  system. .. .Some  insurance 
against  failure  would  be  provided  by  requiring  continuation  of  the  existing 
OER  system  during  the  test  period. 

If  tests  are  approved,  a  test  group  for  each  alternative  should  be 
set  up.  Composition  should  include  a  general  officer  head  and  full  time 
personnel  for  development  of  procedures  and  test  implementation  from  AFDPXO, 
AFDPMAJA,  and  AFHRL. 

SUMMARY  OF  RECOMMENDATIONS 

1.  Standardize  approach  for  developing  job  descriptions.  (Hq  USAF 
action) . 

2.  Require  definition  of  job  requirement  factors  using  on-the-shelf 
AFHRL  methodology.  Forward  with  description  through  indorser  to  assign¬ 
ment  and  promotion  activities  (Hq  USAF  action  -  AFHRL  update). 

3.  Develop  career  field  oriented  job  performance  evaluation  method¬ 
ology.  Output  goes  through  indorser  to  assignment  activity.  Ratee  counseling 
is  based  on  this  visible  form  (AFHRL  action,  after  USAF  concurrence). 

4.  Develop  methodology  to  give  rater /ratee  feedback  on  recent  promotion 
potential  ratings  (AFHRL/AFDPMAJA  (MPC)  actions,  after  USAF  concurrence). 

5.  Define  size  of  ratee  pools  under  different  pool  formulation  rules 
(AFHRL  action). 

6.  Select  short-run  system  modifications  from  alternatives  provided 
on  page  10  and  in  inclosures,  as  necessary  (Hq  USAF  action). 
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7.  Select  methodology  for  field  testing  PAT  and  FRESCA  alternatives 
(Hq  USAF). 

8.  Field  test  and  evaluate  PAT  alternative  (Hq  USAF,  AFHRL,  MAJCOM). 

9.  Field  test  and  evaluate  FRESCA  alternative  (Hq  USAF,  AFHRL, 


MAJCOM) . 

10.  Modify  tested  methodologies  for  USAF-wide  implementation  (AFHRL). 


While  precise 

R&D  COST  ESTIMATES 

estimation  of  costs  associated  with 

these 

recommendations 

is  not  yet  possible 

,  the  following  provides 

indication 

of  magnitudes: 

Recommendation 

Organic  AFHRL 
Many ears 

Contract 
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R&D 
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USAF  EVALUATION  SYSTEMS  WORKSHOP 
El  Tropicano  Hotel 
San  Antonio,  Texas 
25-29  January  1971 

Roster  of  Attendees  and  Participants 


Mr  Charles  H  Anderson 

Program  Director,  Personnel  Measurement, 
Research  &  Development 
Civil  Service  Commission  (PSP) 

Washington,  DC 
202  632-7610 

Dr  Richard  Arvey 
Personnel  Decisions 
University  of  Minnesota 
Minneapolis,  Minnesota  55404 
612  336-5621 

Sqd  Ldr  D.  W.  R.  Barrett 
Royal  Air  Force 
Assigned:  USAFMPC  (DPMRA) 

Randolph  AFB ,  Texas  78148 
Autovon  487-2984/3363 

Dr  Robert  Bolda 

R&D  for  Salary  Personnel  Activity 
Personnel  Staff,  General  Motors 
222  GM  Bldg 
Detroit,  Michigan 
313  556-2596 

Dr  Robert  Bottenberg 

Chief,  Computer  &  Management  Sciences  Branch 
Personnel  Division,  AFHRL 
Lackland  AFB,  Texas  78236 
Autovon  473-3841 

Dr  J.  W.  Bowles 
Deputy  Chief 

Personnel  Division,  AFHRL 
Lackland  AFB,  Texas  78236 
Autovon  473-3387 

Dr  Leland  Brokaw 

Computer  &  Management  Sciences  Branch 
Personnel  Division,  AFHRL 
Lackland  AFB,  Texas  78236 
Autovon  473-3843 


Dr  Raymond  E  Christal 

Chief,  Occupational  &  Career  Development  Branch 
Personnel  Division,  AFHRL 
Lackland  AFB,  Texas  78236 
Autovon  473-2807 

Mr  Joseph  Cowan 

Office  of  Personnel  &  Research  Branch 

US  Coast  Guard 

400  Seventh  Street 

Washington,  DC  20591 

202  426-2158/0890 

LtCol  Francis  W  Craig 

Directorate  of  Military  Personnel  Policies 

Hq  Dept  of  the  Army 

Rm  2E749,  Pentagon 

Washington,  DC  20310 

Autovon  227-3315 

Dr  Roy  A  DeGaugh 

Deputy  Chief  of  Operations 

Hq  AFHRL 

Brooks  AFB,  Texas  78235 
536-3841 

Dr  Henry  Duel 

Chief  of  Personnel  Research 
Directorate  of  Civilian  Personnel 
Hq  USAF 

Washington,  DC  20330 
223-5297 

Dr  Victor  Fields 

Director,  Personnel  &  Training  Research  Programs 

Office  of  Naval  Research 

800  N.  Qunicy  Street 

Arlington,  Virginia  22217 

202  692-4504  Autovon  222-4504 

Wing  Cdr  Ray  Funnell 
Royal  Australian  A.F. 

Assigned:  USAFMPC  (DMRC) 

Randolph  AFB,  Texas  78148 
Autovon  487-5851 


Col  Marvin  Grunzke 

Chief,  Personnel  Research  &  Analysis  Division  (DCS/P) 
Hq  USAF  (AF/DPXY) 

Washington,  DC  20330 
Autovon  227-0452/0453 

Dr  Clifford  Hahn 
American  Institute  for  Research 
Silver  Springs,  Maryland  20910 
301  587-8201 

LtCol  Joe  T  Hazel 
Project  Officer 
Personnel  Division,  AFHRL 
Lackland  AFB ,  Texas  78236 
Autovon  473-3964 

Capt  Bill  Hoadley 
USAFMPC  ( DPMY ) 

Randolph  AFB,  Texas  78148 
Autovon  487-5624 

Capt  Robert  Howen 
Hq  USAF  (DPXO) 

Washington,  DC  20330 
Autovon  227-5170/5154 

Dr  Charles  E  Hutchinson 
Hq  USAF  (AFOSR) 

1400  Wilson  Blvd 
Arlington,  Virginia  22209 
Autovon  224-5045 

Dr  Ed  Locke 

Department  of  Business  Administration 
University  of  Maryland 
College  Park,  Maryland  20740 
301  454-4721/474-8857 

Col  Jim  Marsh 
Hq  USMC 

Arlington  Annex 
Washington,  DC  20380 
Autovon  224-2074 


Dr  Ernest  McCormick 
Department  of  Psychology 
Purdue  University 
Lafayette,  Indiana  46207 
317  749-3551/743-2396 

Dr  William  McGehee 

Director,  Personnel  Research  &  Training 
Fieldcrest  Mills,  Inc. 

Eden,  North  Carolina  27228 
919  623-2123/627-7744 

Maj  Fred  Nordhauser 
Econometrics  Section 
Personnel  Division,  AFHRL 
Lackland  AFB,  Texas  78236 
Autovon  473-4107 

Cdr  F.  S.  Paine,  Jr. 

Bureau  of  Naval  Personnel 
Fitness  Reports  Branch 
Washington,  DC  20370 
Autovon  224-1124 

Col  George  K  Patterson 
Commander ,  AFHRL 
Hq  AFHRL  (CC) 

Brooks  AFB,  Texas  78235 
536-3607 

Mr  David  Robertson 

US  Naval  Personnel  Research  Activity 
San  Diego,  California  92152 
Autovon  952-6122 

LtCol  William  B  Savage 
Selection  Board  Secretariat 
USAFMPC  (DPMAJA) 

Randolph  AFB,  Texas  78148 
Autovon  487-4901/02/03 

Col  James  J  Shepard 
Chief,  Personnel  Division 
Personnel  Division,  AFHRL 
Lackland  AFB,  Texas  78236 
Autovon  473-3385 


Dr  Richard  Smith 

Industrial  Engineering  Department 
Arizona  State  University 
Tempe,  Arizona  85281 
602  965-3610/965-9011/969-3511 

Dr  Robert  Smith 

Department  of  Management 

Kent  State  University 

Kent,  Ohio  44240 

216  672-3080/672-2121/673-1783 

Maj  Henry  Taylor 
Hq  AFSC  (DLX) 

Andrews  AFB 
Washington,  DC  20331 
Autovon  858-4700 

Dr  Ray  Waldkoetter 
Research  &  Analysis  Office 
US  Army  Enlisted  Evaluation  Center 
Ft.  Benjamin  Harrison,  Indiana  46249 
Autovon  239-3621/3622 

LtCol  Gene  A  Weaver 
Classification  &  Stds  Division 
Hq  US  Army  (DCS/P) 

Washington,  DC  20330 
Autovon  227-5220 

Maj  Robert  E  Wilkinson 
OER/Promotion  Systems  Section 
Personnel  Division,  AFHRL 
Lackland  AFB,  Texas  78236 
Autovon  473-3966 

Dr  Virginia  Zackert 
Learning  Materials  Division 
Medical  College  of  Georgia 
Augusta,  Georgia  30902 
404  724-7111 
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USAF  Officer  Evaluation  Workshop 
25-29  January  1971 


Briefings  presented  by  US  Air  Force,  Army,  Navy,  Marine 
Corps,  Coast  Guard,  RAAF/RAF ,  Industry,  and  Civil  Service 
Representatives . 

Tab  1.  Overview  and  Purpose  of  Workshop 
Col  James  Shepard 
Chief,  Personnel  Division,  AFHRL 

Tab  2.  USAF  Officer  Evaluation  System  Today  & 
Proposed  Changes 
Capt  Robert  Howen 
Hq  USAF  (DPXO) 

Tab  3.  OER  in  Selection  and  Promotion  (Temporary) 
Systems  Procedures 
LtCol  William  Savage 
USAFMPC  (DPMAJA) 

Tab  4.  Use  of  OER  in  Assignment  Decisions  and 
Career  Counseling 
Capt  Bill  Hoadley 
USAFMPC  (DPMY) 

Tab  5.  US  Army  Officer  Evaluation  System 
LtCol  Francis  W  Craig 
Hq  Dept  of  Army 

Tab  6.  US  Navy  Officer  Evaluation  System 
Cdr  F.  S.  Paine,  Jr. 

Bureau  of  Naval  Personnel 

Tab  7.  US  Marine  Corps 

Officer  Evaluation  System 
Col  Jim  Marsh 
Hq  USMC 

Tab  8.  US  Coast  Guard  Officer  Evaluation  System 
Mr  Joseph  Cowan 
US  Coast  Guard 


Tab  9 . 


Tab  10. 


Tab  11. 


Tab  12. 


Tab  13. 


The  RAAF  Confidential  Reporting  System 
Wing  Cdr  R.  G.  Funnell,  RAAF 
(Discussion) 

RAAF  and  RAF  Officer  Evaluation  System 
Wing  Cdr  R.  G.  Funnell,  RAAF 
Sqdn  Ldr  D.  W.  R.  Barrett,  RAF 

IBM  Executive  Evaluation  System 
Presented  by: 

Major  Fred  Nordhauser 
Personnel  Division,  AFHRL 
(Under  auspices  of  Mr  P.  S.  Buttress,  IBM 
Harrison,  New  York) 

General  Motors  Executive  Evaluation  System 
Presented  by: 

Major  Robert  Wilkinson 
Personnel  Division,  AFHRL 
(Under  auspices  of  Mr  Robert  Chandler  and 
Mr  Robert  Bolda,  General  Motors  Corp, 
Dearborn,  Michigan) 

J.  C.  Penny  Executive  Evaluation  System 
Dr  Richard  Arvey 
Personnel  Decisions,  Inc. 

Civil  Service  Executive  Evaluation  System 
Dr  Henry  Duel 
Hq  USAF  (AF/DPC) 
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USAF  Officer  Evaluation  System  Workshop 
El  Tropicano  Hotel 
San  Antonio,  Texas 
25-29  January  1971 


Briefing  Schedule  and  Agenda 
25  January  1971  (First  Day) 

0845-0910  Welcome  and  introduction 
Colonel  George  Patterson 
Commander ,  AFHRL 

0910-1000  Overview  and  Purpose  of  Workshop 
Colonel  James  J  Shepard 
Chief,  Personnel  Division,  AFHRL 

1000-1020  Break 

1020-1135  USAF  Officer  Evaluation  System, 

Today  and  Proposed  Changes 
Captain  Robert  Howen 
Hq  USAF  (DPXO) 

1135-1245  OER  in  Selection  and  Promotion  (temporary) 
Systems  Procedures 
LtCol  William  Savage 
USAFMPC  (DPMAJA) 

1245-1345  Lunch 

1345-1430  Use  of  OER  in  Assignment 

Decisions  and  Career  Counseling 
•  Major  Donald  C  Metz 
USAFMPC  (DPMY) 

1430-1520  US  Army  Officer  Evaluation  System 
LtCol  Francis  W  Craig 
Hq  Department  of  the  Army 

Directorate  of  Military  Personnel  Policies 

1520-1535  Break 

1535-1630  US  Navy  Officer  Evaluation  System 
Commander  F.  S.  Paine,  Jr. 

Bureau  of  Navy  Personnel 
Fitness  Reports  Branch 


1630-1645 


1800- 

26  January 
0830-0915 

0915-1000 

1000-1015 

1015-1100 


1100-1130 

1245-1315 


Inf ormation/Comments  for  second  day 
Colonel  Shepard 
LtCol  Hazel 

Cocktails  and  dinner 
Lackland  AFB  Officers  Club 

1971  (Second  Day) 

US  Marine  Corps  Officer  Evaluation  System 
Colonel  William  Marsh 
Hq  USMC  Arlington  Annex 

US  Coast  Guard  Officer  Evaluation  System 
Mr  Joseph  Cowan 

US  Coast  Guard  Office  of  Pers  Rsch  Branch 
Break 

Royal  Australian  Air  Force  Officer  Evaluation 
System 

Wing  Commander  Ray  Funnell 
Royal  Australian  Air  Force 
Assigned:  USAFMPC  ( DMRC ) 

Open  Discussion 

RAAF  and  RAF  Officer  Evaluation  System 
Wing  Commander  Ray  Funnell,  RAAF,  and 
Sqd  Leader  D.  W.  R.  Barrett,  RAF 

IBM  Executive  Evaluation  System 
Major  Fred  Nordhauser 
Personnel  Division,  AFHRL 
(Under  auspices  of  Mr  P.  S.  Buttress, 

IBM,  Harrison,  New  York) 

General  Motors  Executive  Evaluation  System 
Major  Robert  Wilkinson 
Personnel  Division,  AFHRL 
(Under  auspices  of  Dr  Robert  Bolda, 

General  Motors  Corp, 

Detroit,  Michigan) 
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1315-1400  J  C  Penny  Executive  Evaluation  System 
Dr  Richard  Arvey 
Personnel  Decisions,  Inc. 

1400-1430  Civil  Service  Executive  Evaluation  System 
Dr  Henry  Duel 
Hq  USAF  (AF/DPC ) 

1430-1445  Break 

1445-1500  Instructions  for  Panels 

Explanation  of  Tasks,  Guidelines, 
Considerations 
Colonel  James  J  Shepard 
Personnel  Division,  AFHRL 

1500-1630  Panels  Assemble 

(Office  Space  A,  B,  C,  D,  and  Room  3064 
Third  Floor) 


27  January  1971  (Third  Day) 

0830-1630  Panel  Development  of  Research  Proposals 
(Office  Space  A,  B,  C,  D,  and  Room  3064) 


28  January  1971  (Fourth  Day) 

0830-1630  Panel  Development  of  Research  Prospects 
Preparation  of  Presentations 
(Office  Space  A,  B,  C,  D,  and  Room  3064) 


29  January  1971  (Fifth  Day) 

0830-1130  Panel  Development  of  Research  Proposals 
Preparation  of  Presentations 

1130-1245  Lunch 

1245-1300  All  attendees  assemble  in  River  Room  (1st  Floor) . 

Information  concerning  Review/Critique  of  Proposals. 
Colonel  James  J  Shepard 
Personnel  Division,  AFHRL 
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1300-1415  Brief  Presentation  of  Proposals 


1415 

1500 


1300  -  Orange  Panel 
1315  -  Crimson  Panel 
1330  -  Maroon  Panel 
1345  -  Red  Panel 
1400  -  Scarlet  Panel 

1500  Panel  Chairmen  and  Executive  Coordinators 
Meeting  (Hidalgo  Room) 

Discussion  -  Workshop  Report  Preparation 
Workshop  Adjourns 
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COMPUTERIZED  EVALUATION  SYSTEM:  A  PROPOSAL 

by 

Lt  Col  Robert  A.  Dineen 
USAF 


10  February  1971 


Personnel  Division 

Air  Force  Human  Resources  Laboratory 
Lackland  Air  Force  Base,  Texas  78236 


Figures : 


1.  Building  the  graph 

2.  Discrimination  as  a  function  of  time 

3.  The  complete  graph 

4.  Example  of  five  2Lts  and  five  Majors 

5.  Example  of  nine  2Lts  and  one  Captain 

6.  Example  of  graph  as  viewed  by  Promotion  Board 

7.  Example  of  graph  with  control  limits 
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Computerized  Evaluation  System:  A  Proposal 


The  concept  of  a  Computerized  Evaluation  System  requires  radically  new 
thinking  on  the  part  of  the  rater,  the  ratee,  and  the  endorser.  It  involves 
logic  which  must  be  transformed  into  a  mathematical  model,  a  model  which 
must  in  turn  be  converted  into  a  computer  program  that  is  discussed  in 
Appendix  A. 

The  Computerized  Evaluation  System  may  appear  to  be  complicated  from  a 
mathematical  point  of  view,  but  the  procedure  to  determine  the  best  and 
worst  officers  must  be  complicated  to  insure  that  raters  will  not  "beat  the 
system"  as  they  have  in  the  past.  Secondly,  it  has  been  historically 
impossible  to  reproduce  the  policy  decisions  of  the  promotion  board,  which 
in  itself  is  a  very  complicated  process.  Therefore,  the  procedure  outlined 
in  this  proposal  is  relatively  simple  compared  to  the  rationale  of  the  overall 
promotion  board  process. 

Suppose  we  continue  this  discussion  and  analyze  the  system  from  an 
operational  point  of  view  with  a  look  at  the  rater.  The  rater  receives  a 
sealed  envelope  from  Military  Personnel  Center.  Inside  the  envelope  are 
twelve  IBM  cards.  The  first  nine  cards  are  prepunched  with  the  names  of  the 
last  nine  officers  rated  by  the  rater.  These  officers  (ratees)  may  not  be  of 
equal  rank  (in  fact,  they  probably  will  not  be),  but  this  is  compensated  for 
and  will  be  explained  later.  If  the  rater  has  not  rated  officers  before, 
blank  cards  are  used  to  represent  officers  of  the  same  rank  as  the  ratee. 

The  rater  rank  orders  the  nine  cards  according  to  whom  he  feels  was  the  best 
officer,  second  best,  third  best,  and  so  on.  He  then  inserts  in  rank  order 
position  the  next  card  for  the  officer  currently  being  evaluated.  To  insure 


that  the  cards  are  not  rearranged  during  processing,  he  writes  on  each  card 
the  rank  number.  The  eleventh  card  is  not  prepunched  and  has  room  for  80 
characters  of  information  about  the  officer.  This  card  corresponds  with  the 
comments  section  on  the  current  OER  form.  The  ratee  may  abbreviate,  condense 
or  print  as  much  as  he  can  about  the  officer  in  these  80  characters  (blank 
spaces  between  words  count  as  a  character) .  The  cards  are  then  taken  to 
the  endorser,  who  examines  the  ranking  of  the  officers.  If  he  approves,  he 
so  indicates  by  adding  his  signature  to  an  IBM  card  (the  twelvth)  previously 
signed  by  the  rater  and  forwards  the  cards  to  MPC.  If  the  endorser  does  not 
approve,  he  so  indicates  and  then  obtains  from  MPC  a  set  of  cards  identifying 
the  last  nine  officers  that  he,  the  endorsing  official,  rated.  The  endorser 
then  ranks  the  ratee  relative  to  these  nine  officers.  Both  sets  of  cards 
are  placed  in  an  envelope  which  is  self-addressed  to  MPC. 

When  the  cards  arrive  at  MPC,  the  envelope  is  opened,  the  information 
on  the  comment  card  is  punched,  and  the  card  is  placed  back  in  the  deck  (see 
Figure  A).  These  cards  are  then  fed  into  the  computer.  The  computer  does 
statistical  analysis  on  each  card;  a  score  is  computed  for  each  officer  and 
is  stored  in  the  computer.  If  a  set  of  cards  is  submitted  by  an  endorser, 
the  overall  value  for  the  ratee  is  adjusted.  Note  that  this  system  requires 
very  little  writing,  is  completely  confidential  while  the  officer  is  being 
graded,  is  efficient,  can  be  started  on  existing  computer  equipment  at  MPC, 
insures  a  spread  among  the  officers,  and  yet  gives  the  officer  visibility 
with  respect  to  his  career  intentions. 

After  the  ranked  cards  have  been  inserted  into  the  computer  and  the 
appropriate  values  computed,  a  printout  is  provided  to  the  rater  and  endorser 
to  insure  that  the  cards  were  "read"  in  the  correct  order.  The  computed 


Worst  officer 


DOE  JOHN  D  337294019 
(RATEE  IDENTIFICATION) 


3.2  -  03  i  DINEEN  360249337 

'  i  i 

1  , 

TAFSCiRANK  ! (RATER  IDENTIFICATION) 
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Figure  A.  Set  of  Cards  as  Received  by  MPC 
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score  will  also  be  furnished  to  the  ratee,  rater  and  endorser. 


If  an  officer  is  discharged  from  the  Air  Force,  his  records  are  removed 
from  the  promotion  file. 

There  are  numerous  combinations  of  rank  that  can  occur  within  these  ten 
cards.  For  example,  we  may  have  a  mixed  population  of  five  2Lts  and  5  Majors 
A  method  was  developed  to  logically  compensate  for  the  apparent  inequities  of 
2Lts  being  compared  with  Majors.  This  method  will  be  discussed  in  the 
Appendices.  The  fact  that  an  officer  may  have  rated  a  person  five  years  ago 
and  since  that  time  has  forgotten  how  the  officer  actually  performed  is  also 
taken  into  account.  Again,  this  will  be  explained  in  the  Appendices. 

Other  officer  combinations  are: 

(1)  Rater  has  never  rated  an  officer  before. 

SOLUTION:  Nine  blank  cards  are  supplied  to  the  rater  and  the 
rater  places  the  ratee  where  he  thinks  the  ratee  would  fit  in  a  population 
of  like  officers. 

(2)  Rater  rates  the  same  officer  again  without  rating  any  other 

officers . 

SOLUTION:  The  rater  receives  eight  blank  cards  plus  two 
prepunched  cards  with  the  ratee' s  name.  The  ratee  is  now  being  compared 
to  how  well  he  did  in  the  present  rating  period  versus  the  last  rating 
period.  This  process  adjusts  the  first  value  that  the  ratee  received. 

When  the  rater  has  only  one  officer  to  compare  over  two  consecutive 
reporting  periods,  the  endorser  is  required  to  submit  a  rank  order  of  officers 
that  the  endorser  has  formerly  rated.  This  procedure  should  decrease  the 
lieklihood  that  the  rater  inflates  the  ratee 's  score  because  the  endorser 
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has  to  consider  officers  that  he  has  rated  in  the  past  along  with  the  ratee 
and  rater.  If  the  rater  has  been  "overselling"  the  ratee,  then  the  ratee 
may  get  ranked  above  the  rater  in  the  endorser's  ranking  selection.  This 
procedure  will  tend  to  brake  the  inflation  process  when  the  rater  has  only 
one  person  to  rate. 

(3)  Rater  has  to  rate  the  same  officer  for  the  third  consecutive 
time  with  no  intervening  officers. 

SOLUTION:  The  rater  receives  seven  blank  cards  plus  three 
prepunched  cards  with  the  ratee 's  name.  The  ratee  is  now  being  compared 
to  how  well  he  did  during  the  present  rating  period  versus  the  last  two 
rating  periods.  The  values  given  during  the  first  two  periods  will  be 
adjusted  as  the  third  value  is  entered  into  the  officer's  record.  The  endorser 
submits  a  set  of  cards.  The  ratee 's  file  now  contains  the  score  of  the  present 
time  period  (which  carries  the  most  weight)  and  the  scores  given  during  the 
two  previous  time  periods  which  are  adjusted  by  formula  2  in  the  Appendix. 
This  formula  considers  the  length  of  the  reporting  period  and  how  long  ago 
the  officer  was  rated  (the  forgetfulness  function). 

(4)  Rater  has  at  least  ten  officers  of  the  same  grade  working  for 

him. 


SOLUTION:  The  rater  simply  ranks  the  officers  as  to  whom  he 
thinks  is  best,  second  best,  and  so  on  until  the  worst  officer. 


When  a  card  is  submitted  on  officers  who  were  formerly  rated,  the  initial 
score  that  the  officer  received  will  be  adjusted.  It  is  important  to 
understand  that  cards  submitted  after  the  initial  rating  only  adjust  the 
appropriate  score  for  that  time  period.  In  other  words,  a  rater  may  have 
second  thoughts  about  the  rating  that  he  gave  to  a  former  officer  and  have  a 
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chance  to  increase  or  decrease  the  original  value.  An  example  would  be  that 
Officer  A's  record  is  influenced  if  he  is  one  of  the  nine  last  officers  ranked 
by  rater  B  and  rater  B  now  sends  in  a  ranking  that  includes  Officer  A.  In 
doing  so,  the  forgetfulness  function  is  mathematically  taken  into  account 
which  "waters  down"  the  impact  of  the  rater  going  back  in  time  to  adjust  the 
score.  This  method  is  explained  in  the  Appendices. 

The  end  result  is  a  series  of  scores  that  form  a  trend  on  how  the 
officer  performed  with  respect  to  other  officers  over  time.  The  trend  is 
used  by  the  promotion  board  to  determine  who  should  be  promoted. 

An  important  ingredient  of  this  system  will  be  the  promotion  board 
which  makes  the  final  decision  as  to  who  is  promoted.  The  Computerized 
Evaluation  System  guarantees  that  both  the  "fastburners"  and  the  "slowburners" 
are  spotlighted.  During  the  transition  period,  the  promotion  board  will  have 
as  a  part  of  the  officer  selection  folder  a  printout  showing  how  the  officer 
was  rated  in  the  computerized  system.  The  80  character  comment  will  be 
included  as  a  part  of  this  printout,  as  will  some  rather  elementary 
statistical  analysis  (see  Appendix  A) .  A  printout  of  the  computer  report 
will  be  sent  to  the  officer  at  the  time  of  his  annual  record  review  so  as 
to  give  him  some  degree  of  visibility.  This  printout  indicates  how  he  stands 
relative  to  his  year  group  (see  figure  7  in  Appendix  A). 

The  next  stage  in  the  development  of  this  system  would  be  to  build  a 
simulation  system  to  test  various  officer  combinations  over  time,  along  with 
other  parameters  of  the  system,  i.e.,  ranking  five  officers  versus  ten,  other 
scales,  etc.  This  could  be  accomplished  by  the  Personnel  Division  within  three 
months . 
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A  test  program  could  be  set  up  with  a  finite  population  (i.e.,  Wing, 
Division,  Command,  etc)  in  order  to  obtain  feedback  from  the  using  commands. 
This  should  take  about  one  year.  The  salable  features  of  this  proposal  are: 

(1)  It  insures  a  spread  among  officers  that  the  rater  has  formerly 
rated.  This  spread  is  more  critical  as  an  officer  has  more  time  as  a 
commissioned  officer.  In  fact,  the  system  has  a  de-inf lationary  effect 
because  an  average  officer  receives  lower  ratings  as  he  obtains  more  time 
in  service.  An  important  feature  of  this  program  is  the  trend  of  scores 
that  an  officer  receives  over  time. 

(2)  It  is  simple  to  operate  and  tamper-proof. 

(3)  The  system  provides  that  the  rating  is  confidential  at  the  time  of 
the  rating,  yet  gives  the  officer  (ratee)  visibility  with  respect  to  officers 
of  his  same  year  group.  The  rating  that  an  officer  receives  will  be  contingent 
upon  the  other  ten  officers  that  he  was  rated  against.  Since  the  ratee  is 
never  aware  of  the  population  of  the  nine  other  officers,  this  is  the  trick 
that  insures  us  that  the  rating  is  confidential  at  the  time  of  the  rating. 

In  other  words ,  the  ratee  who  receives  a  low  grade  may  rationalize  that  he 
was  working  with  superior  officers  and  that  the  rater  was  "forced"  to  stack 
the  ratee  in  a  low  position. 

(4)  Maximum  use  is  made  of  algorithms,  statistics,  and  computers. 

(5)  The  promotion  board  still  makes  the  final  decision  based  upon  a 
computer  printout  of: 

(a)  pertinent  Uniform  Officer  information 

(b)  graphical  trend  analysis  of  how  well  the  officer  compared  with 
other  officers  vying  for  promotion 

(c)  short,  pertinent  comments  about  the  officer  correlated  to 
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the  scores  on  the  trend  line  of  each  officer. 

(6)  It  has  several  "built-in"  quality  control  devices  to  assist  the 
promotion  board  in  analyzing  a  score  that  is  out  of  statistical  limits. 

(7)  A  similar  system  could  be  initiated  to  evaluate  airmen. 

The  average  group  that  raters  rated  during  CY1970  is  as  follows: 


OFFICERS  RATED  DURING  CY1970 


Ratees 

2/Lt 

1/Lt 

IWMHhi 

icsm 

L/C 

Col 

Total 

2/Lt 

1.3 

n 

1/Lt 

.64 

.75 

i 

H 

Capt 

.34 

.93 

1.05 

2.3 

Raters 

Maj 

.16 

.73 

2.03 

.52 

3.5 

L/C 

.11 

.43 

1.43 

1.39 

.47 

3.8 

Col 

.09 

.27 

.97 

1.1 

1.78 

1.78 

4.9 

For  example,  during  CY1970  Captains  on  the  average  rated  0.34  2/Lts, 
0.93  1/Lts,  and  1.05  Captains.  Based  on  these  statistics,  Captains  rated  an 
average  of  2.3  officers.  Therefore,  it  would  take  4.3  years  before  Captains 
would  have  ten  officers  in  their  unique  population  versus  a  little  more  than 
two  years  for  Colonels.  Hence,  part  of  the  analysis  of  this  proposal  should 
include  the  proper  population  size  for  each  rater  grade. 
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APPENDIX  A 


To  keep  the  mathematics  at  a  talking  level  without  going  into  the 
details  of  where  the  numbers  come  from,  suppose  we  begin  this  discussion 
with  a  graphical  analysis  of  how  we  develop  the  scores.  The  left  hand  side 
of  the  graph  (the  ordinate)  can  be  looked  upon  as  a  promotion  score  of  sorts 
and  is  scaled  from  0  to  100.  The  bottom  of  the  graph  (the  abscissa)  is 
scaled  from  0  to  9  and  corresponds  with  the  rank  order  of  cards  as  they 
are  read  by  the  computer  (0  represents  the  best  officer) .  Now,  suppose  for 
the  moment  that  our  population  is  restricted  to  2Lts  and  that  we  draw  a  line 
connecting  the  upper  left  corner  of  the  graph  with  the  lower  right  corner  of 
the  graph  as  in  Figure  1.  If  we  then  locate  on  the  abscissa  the  relative 
position  of  an  officer  (relative,  that  is,  to  the  other  nine  officers 
against  whom  he  was  rated),  project  this  rating  onto  the  promotion  score 
scale,  and  repeat  this  operation  for  each  officer  in  the  population,  we  will 
get  a  distribution  of  officers  along  the  entire  scale.  The  best  2Lts  would 
have  a  promotion  score  of  100  and  the  worst  would  have  a  score  of  zero.  While 
this  would  give  maximum  discrimination  between  the  2Lts,  it  would  be  of  little 
or  no  value  because  virtually  all  2Lts  and  lLts  are  100%  qualified  for 
promotion.  They  should  all,  therefore,  be  clustered  around  the  top  end  of 
the  promotion  score  scale.  We  can  achieve  this  more  clustered  distribution 
by  reducing  the  slope  of  our  projection  line. 

In  Figure  2,  we  have  decreased  the  slope  of  the  projection  line  for  2Lts 
and  have  introduced  lines  for  each  of  the  other  ranks.  As  rank  increases, 
so  increases  our  desire  to  discriminate  more  carefully  between  the  officers 
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eligible  for  promotion.  The  slope  of  the  projection  line  for  2Lts  is 
shallow  as  we  are  little  concerned  with  discriminating  between  officers 
in  this  group.  The  slope  for  Lt  Cols  with  28  years  of  service  is  quite 
steep  in  order  to  insure  maximum  discrimination.  A  discriminating 
factor  between  officers  of  the  same  rank  is  time  in  service.  That  is, 
eligibility  can  be  related  to  time  in  service  and  vice  versa.  Total  active 
federal  commissioned  service  (TAFCS)  time  is  included  as  a  third  scale  in 
Figure  2.  This  scale  has  a  range  of  0  to  28  years.  Twenty-eight  years 
was  selected  as  the  bottom  end  of  the  scale  because  a  Lt  Col  who  is  not 
promoted  to  Colonel  at  28  years  is  forced  into  retirement.  Notice  also 
that  the  values  on  the  right  hand  side  are  not  equally  spaced.  The  range 
from  0  to  6  years  service  time  extends  over  more  than  fifty  points  on  the 
promotion  score  scale;  the  range  of  6  to  28  years  covers  slightly  less  than 
a  fifty  point  spread.  This  approach  gives  more  of  a  spread  in  the  early  years 
of  the  officer's  career  development.  Conversely,  there  should  be  little 
distinction  between  a  Lt  Col  of  24  to  28  years  of  service. 


There  is  one  more  factor  needed  to  complete  this  graph.  That  is,  we 
need  a  way  to  spotlight  the  "f astburners . "  This  is  done  by  incorporating  a 
bend  near  the  upper  left  hand  side  of  the  lines  that  increases  as  the  TAFCS 
increases.  This  curve  represents  a  negative  exponential  curve  of  the  following 
form: 

F(x)  =  100  e-ax  (1) 


where , 

a  =  constant  of  proportionality 

x  =  total  active  federal  commissioned  service  in  years. 
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The  completed  graph  is  presented  in  Figure  3.  The  mathematics  and 
computer  program  used  to  generate  the  promotion  score  values  are  in 
Appendix  C. 

An  example  situation  might  be  useful  to  help  clarify  the  logic  of  this 
graph.  Let  us  take  a  simple  case  first  where  we  have  all  2Lts  with  six 
months  of  service.  Each  would  be  ranked  by  his  rater  as  best  or  worst  (0 
to  9)  relative  to  his  peers,  and  each  would  be  assigned  an  appropriate  score. 
This  would  begin  early  in  their  career  the  discrimination  between  who  was  the 
best  and  who  was  the  worst. 

As  time  progresses  and  they  obtain  more  time  in  service,  the  lines  slant 
increasingly  more  downward  and  an  increasingly  greater  degree  of  discrimination 
between  officers  is  possible.  Eventually,  the  best  officer  will  be  identified 
as  will  the  worst. 

Let  us  take  an  example  where  we  have  five  2Lts  and  5  Majors.  The  trick 
is  to  find  a  base  line  where  we  can  equitably  measure  the  differences  between 
Lts  and  Majors  when  they  are  in  a  mixed  population.  A  base  line  of  Lts  and 
Majors  will  fall  somewhere  between  the  Lt  line  and  the  Maj  line.  The  base  line 
is  computed  from  the  average  years  of  commissioned  service  for  all  ten  officers 
Using  this  base  line,  as  in  Figure  4,  we  can  see  that  a  value  for  one  Maj  and 
the  appropriate  base  line  value  is  computed.  The  absolute  difference  between 
the  two  values  is  found  and  subtracted  from  the  Major's  score.  This  operation 
tends  to  negate  the  differences  between  Majors  and  2Lts.  Conversely,  for  the 
2Lts  a  value  is  computed  and  a  corresponding  base  line  value  is  computed.  The 
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ARBITRARY  SCALE 
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Figure  3 


The  Complete  Graph 


difference  between  these  two  values  is  calculated  and  is  added  to  the  2Lt 
score  (see  Figure  4) . 

One  final  example  will  be  used  to  illustrate  the  point.  Let  us  take  a 
population  of  nine  2Lts  and  one  Capt.  A  base  line  is  mathematically 
computed  based  on  the  average  TAFCS  and  the  appropriate  differences  are  found. 
Now,  if  the  Capt  were  in  fact  less  qualified  for  promotion  than  the  first 
two  2Lts ,  this  graphical  technique  highlights  the  fact  that  the  captain  is 
less  likely  to  be  promoted  than  the  two  2Lts  (see  Figure  5) . 

The  next  problem  to  be  overcome  involves  a  rater  who  must  rank  an  officer 
who  worked  for  him  five  years  ago.  As  time  passes,  the  rater  has  a  tendency 
to  forget  how  this  officer  performed  relative  to  the  other  officers  whom  he 
has  evaluated.  Also,  he  may  have  supervised  the  ratee  for  a  relatively  short 
period  of  time  and  this  also  must  be  taken  into  account. 

Both  problems  are  solved  by  weighting  the  score  by  the  length  of  time 
that  the  ratee  was  observed  by  the  rater  and  by  the  time  that  has  lapsed 
since  the  ratee  was  last  managed  by  the  rater.  Stated  mathematically 


WSCR  =  L 

all  i 


(TOBSj  *  TLASTj)  *  SCRj 

E  (TOBS.  *  TLASTj ) 
all  i 


(2) 


where , 

WSCR  =  weighted  score,  which  are  the  values  computed  for  the  promotion 
board 

TOBS^  =  time  observed  (a  linear  function) 

TLASTj  =  time  since  last  observed  (  a  negative  exponential  function) 
which  is  explained  in  Appendix  B 
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SCR  =  the  adjusted  score  that  an  officer  receives  based  on  his 

ranking  order  and  time  in  service  relative  to  the  population 
of  ten  officers. 

Z  =  sum  over  all  values  that  the  officer  received  during  any  one 

all  i 

period  of  time. 

Therefore,  in  computing  an  individual's  final  score,  two  factors  are 
taken  into  consideration: 

(1)  time  that  the  officer  worked  for  the  rater;  and 

(2)  the  lapsed  time  since  he  last  worked  for  him. 

NOTE:  That  the  rater  may  have  the  opportunity  to  go  back  in  time  and  adjust 

the  original  score  that  he  gave  the  ratee. 

Now,  how  do  we  protect  the  individual  officer  as  he  is  rated?  Does  he 
have  the  opportunity  to  administratively  remove  OERs?  The  answer  is  "No!" 
What  will  happen  is  that  all  these  values  form  a  trend  such  as  Figure  6. 

If  the  person  is  an  average  officer,  he  will  have  a  downward  slope  in  the 
graph.  A  statistical  quality  control  device  will  be  built  into  the  system 
such  that  if  an  OER  score  falls  outside  of  some  statistically  controlled 
limits  -  that  value  would  not  be  considered.  In  other  words,  if  this  officer 
was  getting  good  OERs  but  suddenly  he  received  a  bad  OER  which  fell  outside 
of  some  specified  statistical  limits,  that  value  would  be  coded  as  such 
and  it  would  stay  in  the  officer’s  file.  In  other  words,  there  is  no  way 
that  an  OER  can  be  administratively  removed.  However,  it  can  be  qualified 
as  to  its  worth. 
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ARBITRARY  SCALE  (SCR) 


0  1  2  3  4  5  6  7  89 

RANKING 


Figure  4.  Example  of  Five  2Lts  and  Five  Majors. 


Score  A:  Officer  who  was  ranked  second  would  receive  this  score  if  he  were 
measured  against  other  officers  with  same  TAFCS. 

Score  B:  Officer  receives  adjusted  score  because  he  was  measured  against  a 
mixed  population  of  Majors  and  2Lts. 

Score  C:  2Lt  who  was  ranked  seventh  received  an  upward  adjusted  score 

because  he  was  measured  against  a  population  of  Majors  and  2Lts. 
The  maximum  score  that  he  could  receive  is  100. 
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ARBITRARY  SCALE  (SCR) 
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Figure  5.  Example  of  nine  2Lts  and  one  Capt 


Score  A:  Value  that  the  officer  would  have  received  if  he  was  better 
that  the  2Lts. 

B:  Value  that  the  officer  received  because  2Lts  were  better 
than  the  Captain. 
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WEIGHTED  SCORE  (WSCR) 


NOTE:  These  values  come  from  Equation  (2). 


Data  points  out  of  statistical 
limits ,  therefore  coded  as  an 
asterisk 


0123456789 
TOTAL  ACTIVE  FEDERAL  COMMISSIONED  SERVICE 

Figure  6.  Example  of  Graph  as  Viewed  by  Promotion  Board 
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WEIGHTED  SCORE  (WSCR) 


Confidence  limits  of  the  population  of  officers  being 
/  /  considered  for  promotion. 


Officers  above  the  dashed 
line  are  considered  for 
promotion  below  the  zone 


Officers  below  the  dashed 
line  have  a  low  probability 
of  being  promoted 


Good 

Probability 
of  Being 
Promoted 


TOTAL  ACTIVE  FEDERAL  COMMISSIONED  SERVICE 


Figure  7.  Example  of  Graph  with  Control  Limits 
As  Viewed  by  the  Promotion  Board 
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The  last  portion  of  this  system  is  a  graph  that  would  be  printed  by  the 
computer  on  each  individual  officer  before  the  information  is  presented  to 
the  Selection  Board  Secretariat.  The  graph,  such  as  in  Figure  7,  would  have 
control  limits  which  show  where  the  population  of  the  group  of  officers  being 
considered  for  selection  should  lie.  If  an  officer’s  trend  line  is  above 
the  line,  he  is  a  superior  officer.  Conversely,  if  he  is  below  the  line,  he 
is  a  poor  performer.  The  asterisks  on  the  graph  indicate  that  the  value  was 
outside  the  control  limits  of  the  individual  officer's  values.  A  'T* 
indicates  a  training  report  was  submitted.  It  is  placed  where  a  forecasting 
routine  computed  what  the  officer's  next  value  should  be. 

Other  products  available  to  the  Promotion  Board  would  be  a  Uniform 
Officer  Record  printout  and  eighty  character  comments  associated  with  each 
score. 
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APPENDIX  B 


Forgetfulness  function 

Given:  We  forget  by  a  negative  exponential  function  of  the  form 


f(x)  =  e-ax 


where,  x  is  time  in  years 

a  is  a  constant  of  proportion 


Assume:  That  at  the  end  of  ten  years  we  remember  15%  of  what  a  rater 

originally  knew  about  a  ratee. 

Find :  a 


Solution: 

0.15  =  e_al° 

In  0.15  =  -10a  In  e 


-1.897  =  -10a 

a  =  1.897/10 
=  0.1897 

f(x)  =  e-°'1897x 
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APPENDIX  C 


Computer  Program  to  compute  values  for  graph. 


-23. 


4 


Example  Problem 


Given : 

An  officer  with  Total  Active  Federal  Commissioned  Service  (TAFCS)  of 
six  years,  is  ranked  fourth  in  a  population  of  ten  officers  and  his  unique 
population  of  ten  officers  has  an  average  TAFCS  of  11.0  years. 


Find: 

His  initial  score  based  on  TAFCS  and  ranked  position  relative  to  his 
population. 


Assume : 

That  the  more  time  an  officer  has  in  service,  the  less  discrimination 
is  given  to  his  performance.  Let  us  say  that  his  performance  is  assigned 
an  arbitrary  value  between  0  and  100,  and  that  the  relationship  between  the 
performance  value  and  the  years  that  he  has  in  service  is  a  negative 
exponential  function  of  the  following  form: 

f (z)  =  100e~&z  (Cl) 


where , 

f(z)  =  an  arbitrary  scale  from  0  to  100 
8  =  constant 
z  =  years  of  service. 

Since  the  definition  of  the  function  on  the  right  hand  side  approaches  zero 
asymptotically,  we  assign  a  value  of  one  on  the  arbitrary  scale  for  an  officer 
with  28  TAFCS  years  of  service  who  is  ranked  last.  We  can  now  state  that: 


f (z)  =  f (28)  =  1  =  100e"g28 


Solving  for  Beta  yields  0.164.  Therefore, 

f (z)  =  100e_0'164z 


(C2) 


Now,  for  any  officer's  TAFCS  who  is  ranked  last,  we  can  determine  his  score 
from  the  above  equation. 

In  order  to  define  the  particular  function  for  an  officer  that  has  a  TAFCS  of 
six  years  and  is  ranked  fourth,  we  first  find  his  value  as  if  he  were  ranked 
last  from  equation  C2  and  set  the  computed  value  equal  to  the  following 
ranking  equation,  or 


f (r)  =  100e-ar 
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(C3) 


where ; 

f(r)  =  the  arbitrary  scale  from  0  to  100 
a  =  constant  to  be  determined 
r  =  the  ranked  position. 

For  our  particular  case,  an  officer  with  six  TAFCS  years  and  ranked  last  (or 
ninth,  on  a  0-9  scale)  receives  a  score  of  37.277.  Stated  mathematically 

f (9)  =  100e_a9  =  37.277 

and  solving  the  equation  for  Alpha,  we  find  that 

a  =  0.011. 

Therefore,  the  equation  which  defines  an  officer  with  six  TAFCS  is  of  the 
following  form: 

f (r)  =  100e-0' 011r  (C4) 

Since  the  particular  officer  was  ranked  fourth,  we  adjust  the  ranking  order 
(because  the  scale  is  from  0  to  9)  and  solve  the  following  equation: 

where ; 

f(r-l)  =  lOOe-0'011^-1) 


f (3)  -  71.97 

We  now  compute  the  constant  associated  with  the  average  TAFCS  of  the  population. 
That  is,  we  use  equation  C2  where  z  is  equal  to  11.0.  This  yields  an  Alpha 
value  of  0.201  and  a  ranking  value  of  54.72.  Computing  the  absolute  difference 
between  the  officer's  score  and  the  population  yields, 

71.97  -  54.72  =  17.25 

By  adding  17.25  onto  the  71.97  value  obtained  when  the  officer  had  six  years 
of  service,  we  compensate  for  an  officer  that  has  less  TAFCS  than  his  population 
that  he  was  judged  against. 

Although  the  system  appears  to  be  complicated,  it  reduces  to  about  six  lines 
of  FORTRAN  statements. 
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ABSTRACT 

This  study  provides  a  refinement  of  the  procedures  previously  developed  for  the 
determination  of  the  appropriate  distribution  of  officer  grades  for  the  Air  Force  to  permit 
application  of  the  same  position  evaluation  procedures  to  individual  officer  positions. 

A  scale  of  benchmark  jobs  with  titles  of  established  successive  levels  of  requirements 
was  used  to  derive  a  set  of  job  requirement  factor  scales.  These  scales  were  applied 
to  1000  Air  Force  officer  position  descriptions  collected  and  previously  applied  in  the 
Officer  Grade  Requirements  Study.  Comparison  of  rating  distributions  based  upon  adjec¬ 
tive  scales  and  the  benchmark  scales  revealed  lower  mean  values,  larger  standard  devia¬ 
tions,  and  superior  zero-order  validity  of  the  ratings  based  on  the  benchmark  scales.  The 
predictive  efficiency  of  optimal  composites  of  the  benchmark  scales  was  equivalent  to 
that  of  the  adjective  scales.  A  set  of  integer  weights  for  use  in  field  application  of  the 
equations  was  derived  without  significant  loss  of  validity. 
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DEVELOPMENT  OF  BENCHMARK  SCALES  FOR  AIR  FORCE 
OFFICER  POSITION  EVALUATION 


I.  INTRODUCTION 

The  United  States  Air  Force,  like  many  industries,  has  a  position  evaluation  program  based 
upon  rating  systems  involving  job  requirement  factors.  Several  methods  of  applying  such  ratings 
have  evolved,  but  most  of  them  pivot  on  some  system  of  point  evaluation  with  reference  to  esta¬ 
blished  jobs.  (Madden  &  Giorgia,  1961) 

The  intervals  of  scales  used  in  description  of  these  requirement  factors  are  variously  defined. 
In  some  scales  simple  adjective  definitions  ranging  from  a  very  small  to  a  very  large  requirement, 
associated  with  a  numeric  value,  are  used;  in  others,  longer  descriptions  attempt  to  illustrate  the 
functions  of  a  factor  at  various  scale  levels.  The  official  Air  Force  procedure  cited  in  AFM  35-2, 
Occupational  Analysis,  dated  10  January  1963,  follows  the  latter  procedure. 

The  Air  Force  has  completed  an  overall  study  of  appropriate  grade  level  for  Air  Force  Offi¬ 
cers.  This  Officer  Grade  Requirements  study  (OGR)  has  been  reported  by  Christal  (1965)  and 
Hazel  (1965).  The  study  involved  ten  job  requirement  factors  rated  on  a  simple  adjective  scale 
and  a  large  number  of  background  variables.  It  was  designed  to  provide  content-based  deter¬ 
mination  of  appropriate  grade  for  officer  jobs,  to  provide  guidance  as  to  the  proper  distribution 
of  grades  throughout  the  Air  Force.  The  current  study  is  intended  to  refine  the  system  until  it 
can  be  applied  to  single  positions  for  manning  document  purposes. 

In  the  OGR  study  a  Headquarters  USAF  board  established  the  proper  grade  for  a  representa¬ 
tive  sample  of  officer  jobs.  Raters  throughout  the  Air  Force  judged  appropriate  levels  on  the  job 
requirement  factors  and  grade  level  appropriate  to  these  jobs.  A  multiple  linear  regression  model 
was  applied  to  determine  the  optimal  variables  for  effective  determination  of  appropriate  grade  as 
established  by  the  board. 

Five  of  the  ten  requirement  factors  for  which  data  were  collected  under  this  study  gained 
places  in  the  final  regression  equation.  This  might  have  been  anticipated  in  the  light  of  Madden’s 
findings  that  appropriate  pay  is  predicted  by  factors  different  from  those  predictive  of  appro¬ 
priate  grade  (1963).  Factors  appropriate  to  both  pay  and  grade  were  included  in  the  Officer 
Grade  Requirements  study,  because  the  capability  to  evaluate  appropriate  pay,  as  well  as  grade, 
was  desired. 

The  ratings  based  on  the  adjective  scales  have  acceptable  validity,  as  demonstrated  by 
Christal’s  positive  findings  in  the  OGR  study;  but  it  is  noted  that  the  distributions  of  ratings 
granted  by  these  scales  are  narrow  of  variance,  with  the  median  and  low  values  of  scales  infre¬ 
quently  used. 

Levels  of  ratings  assigned  to  a  given  job  are  subject  to  the  context  in  which  ratings  are 
made.  As  reported  by  Madden  in  I960,  the  same  position  which  is  evaluated  in  a  context  of 
"low”  positions  as  "high,”  in  a  context  of  "high”  positions,  will  be  rated  "low.”  Only  by 
rating  in  a  mixed  context  can  stable  ratings  of  appropriate  level  be  achieved. 

The  present  study  is  based  upon  the  hypothesis  that  application  of  a  factor  rating  scale 
^in  which  the  levels  were  characterized  by  the  titles  of  Air  Force  jobs  familiar  to  rating  officers 
would  both  broaden  the  distributions  of  ratings  given  and  eliminate  the  context  effects. 
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II.  PROCEDURE 


This  study  proceeds  through  three  major  phases  which  will  be  detailed  below.  The  first 
of  these  encompasses  the  development  of  benchmark  scales,  composed  of  job  titles  on  which 
there  is  maximal  agreement  among  Air  Force  officers  as  to  level  of  requirement  for  each  of  the 
ten  factors.  The  second  step  involves  the  collection  of  job  evaluations  in  terms  of  those  factors 
for  a  sizeable  number  of  jobs  on  which  firm  information  as  to  appropriate  grade  and  relative 
factor  ratings  is  available.  The  third  step  involves  digestion  of  the  rating  data  to  establish 
its  predictive  efficiency  for  the  appropriate  grade  levels  and  to  establish  the  procedures  that 
might  be  used  in  the  field  for  determination  of  proper  grades  for  officer  positions. 

Identification  of  Benchmark  Position  Titles 

The  identification  of  job  titles  on  which  substantial  agreement  exists  among  Air  Force 
officers  as  to  level  on  each  of  the  ten  job  requirement  factors  used  in  the  OGR  study  was  the 
first  step  of  this  research.  Data  existed  as  to  field  ratings  by  officers  in  grade  of  major  or  lieu¬ 
tenant  colonel  for  each  of  a  total  of  11,789  jobs.  These  jobs,  on  the  basis  of  descriptions  sub¬ 
mitted  by  incumbents,  had  been  evaluated  by  five  to  seven  rating  officers  each.  Lists  were 
obtained  which  indicated  the  mean  and  standard  deviations  of  ratings  given  each  job  on  each 
factor. 

It  had  been  decided  that  the  final  scales  would  involve  nine  levels;  for  this  reason  the 
list  of  11,789  jobs  for  each  factor  was  divided  into  nine  equal  sections,  ordered  from  high  to 
low  on  mean  ratings  given.  Eleven  job  titles  were  selected  from  each  of  these  nine  groups  of 
titles.  They  were  selected  to  have  minimum  standard  deviations  and  were  so  selected  as  to 
emphasize  the  range  of  ratings  given.  That  is,  for  the  fifth,  or  median  group,  the  eleven  titles 
were  selected  as  near  center  of  the  range  of  means  involved  as  possible;  for  groups  other 
than  the  fifth,  titles  were  chosen  from  the  portion  of  the  range  remote  from  the  median.  This 
had  the  effect  of  identifying  the  highest  possible  mean  rating  for  titles  for  use  as  the  top  or 
"9”  category,  and  the  lowest  possible  mean  rating  for  titles  for  use  as  the  lowest  or  "1” 
category. 

Each  of  the  99  titles  accompanied  by  the  organizational  designation  and  geographical 
location  for  each  factor  was  reproduced  on  a  manila  card.  Decks  of  such  99  titles  were  sent 
to  volunteer  rating  officers  in  the  grade  of  major  and  lieutenant  colonel,  whose  task  was  to 
arrange  the  titles  in  order  in  terms  of  their  requirement  for  the  factor  for  which  they  had  been 
identified.  Each  rating  officer  was  concerned  with  one  factor  for  a  single  group  of  99  job 
titles.  The  ratings  of  job  titles  were  required  to  establish  the  value  of  the  title  in  the  absence 
of  its  detailed  job  description. 

At  such  time  as  15  ratings  for  a  given  factor  had  been  collected,  the  intraclass  correla¬ 
tion  of  the  ratings  across  the  15  raters  for  the  99  jobs  was  determined  (Haggard,  1958,  Chapter 
VIII).  When  a  value  in  excess  of  .92  was  obtained,  the  ratings  were  accepted  as  sufficiently 
stable  for  use.  If  lesser  reliability  was  indicated,  additional  cases  were  collected.  The 
addition  of  cases  for  some  factors  did  not  improve  the  reliabilities  obtained;  in  these  cases 
application  of  a  simple  clustering  technique  revealed  the  presence  of  divergent  policies  on  the 
part  of  rating  officers.  In  each  instance  an  "aircrew”  bias  was  present,  in  which  the  rankings 
given  by  rated  personnel  differed  from  rankings  given  by  non-rated  personnel. 

Tests  for  bias  in  the  Officer  Grade  Requirements  study  previously  cited  had  revealed  no 
systematic  relationship  between  the  aircrew  or  non-aircrew  category  of  rating  officer  and  the 
final  grade  ratings  made.  On  this  basis  it  was  judged  that  policy  differences  in  factor  ratings 
should  be  resolved  by  acceptance  of  the  policy  which  was  most  nearly  in  accord  with  the  factor 
level  determinations  of  the  OGR  study.  The  rank  ordering  on  the  factor  in  question  for  the  set 
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of  99  jobs  in  the  original  11,789  job  sample  was  introduced  into  the  data  received  from  the  field 
raters  and  the  clustering  accomplished.  In  each  such  instance  the  OGR  ordering  was  highly 
related  to  one  of  the  policy  clusters,  and  that  cluster  was  chosen  to  serve  as  the  base  for  the 
final  ordering  of  jobs  for  the  scale  definitions.  When  this  action  was  taken,  and  the  intraclass 
correlation  computed  across  the  cases  so  identified,  the  reliability  value  for  the  ranking  of  all 
factors  was  raised  above  the  .92  level.  The  values  obtained  appear  in  Table  1. 


Table  1.  Intraclass  Correlations  of  Factor  Ratings 
Across  99  Selected  Job  Titles 


Factor 

Number 
of  Ratings 

Intraclass 

Correlation 

1 

Formal  Education 

15 

.92 

2 

Special  Training  &  Work  Experience 

9 

.96 

3 

Working  Conditions 

15 

.92 

4 

Originality,  Ingenuity,  &  Creativeness 

10 

.94 

5 

Communication  Skills 

11 

.94 

6 

Interpersonal  Skills 

11 

.94 

7 

Judgment  &  Decision  Making 

11 

.95 

8 

Planning 

15 

.92 

9 

Management 

9 

.95 

10 

Risk 

15 

.96 

Each  set  of  99  job  titles  as  ordered  by  the  mean  ranking  was  subdivided  into  9  groups  of 
11  titles  each.  The  three  jobs  with  ratings  of  smallest  variance  in  each  subgroup  were  chosen 
for  inclusion  in  the  list  of  27  titles  composing  the  benchmark  list  for  each  factor.  When  the  27 
had  been  so  identified,  the  derived  lists  were  reviewed.  In  a  number  of  cases  titles  of  slightly 
greater  variance  were  substituted  for  titles  identified  as  less  variant. 

This  was  done  primarily  to  reduce  the  redundance  of  titles  in  the  scale— certain  similar 
job  titles  were  given  identical  ratings;  different  kinds  of  jobs  were  inserted  to  make  the  lists 
more  heterogeneous.  In  certain  factor  areas  it  was  not  possible  to  eliminate  all  such  redun¬ 
dance— there  is  no  doubt,  for  instance,  of  the  universal  agreement  that  certain  types  of  pilot 
jobs  belong  at  the  top  of  the  "Risk”  scale. 

Although  ordering  data  for  the  job  titles  had  been  collected  in  a  context  of  organizational 
identification  and  geographical  address,  it  was  discovered  that  certain  security  regulations  would 
be  involved  in  their  publication  as  such.  For  this  reason  the  specific  organizational  identifica¬ 
tion  was  altered  to  indicate  a  type  of  organization,  and  the  geographical  address  either  deleted 
or  changed  to  a  general  designator,  such  as  "overseas”  when  the  final  benchmark  lists  were  pre¬ 
pared.  The  scales  and  instructions  for  the  ten  factors  appear  as  Appendix  I. 

Collection  of  Job  Evaluations 

The  benchmark  lists  were  circulated  to  volunteer  rating  officers  in  the  grade  of  major  and 
lieutenant  colonel,  each  of  whom  evaluated  five  job  descriptions,  selected  from  the  criterion 
sample  of  3,575  jobs  evaluated  in  the  Officer  Grade  Requirements  study.  One  thousand  such 
jobs  were  evaluated,  with  circulation  to  seven  or  more  rating  officers  for  each  packet  of  five, 
and  ratings  were  obtained  for  nearly  all  the  jobs  from  not  less  than  five  raters  each. 
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The  thousand  jobs  used  were  selected  from  a  subsample  of  1,700  jobs  which  had  been 
re-rated  in  the  second  phase  of  the  Officer  Grade  Requirements  study,  when  the  base  of  3,575 
jobs  was  broadened  to  11,789  jobs  as  a  suitable  sample  for  making  Air  Force-wide  projections 
(Christal,  1965). 

This  extensive  information  from  three  sets  of  rating  data  permitted  the  establishment  of 
stability  over  time  of  ratings  on  the  ten  factors,  albeit  on  the  most  recent  study  the  scale  for 
the  ratings  had  been  altered. 

The  rate-re-rate  correlations  across  the  three  rating  occasions,  with  the  means  and  stand¬ 
ard  deviations  for  the  ten  factors  and  the  grade  level  are  presented  in  Table  2.  These  correlations 
are  indicators  of  stability  over  time  and  are  found  to  be  of  the  order  usually  associated  with 
aptitude  tests,  ranging  from  .60  to  .92  among  the  factors,  with  a  median  near  .75.  The  grade 
ratings  were  less  variable,  with  the  lowest  value  at  .88  and  the  highest  at  .91.  These  values 
tend  to  agree  with  the  consistency  suggested  by  the  intraclass  correlations  derived  at  the  time 
the  scales  were  assembled  and  are  as  stable  as  many  personnel  management  measurements. 

Inspection  of  the  distribution  statistics  in  Table  2  shows  that  most  of  the  mean  values  for 
the  benchmark  scales  applied  to  the  factors  were  lower  than  the  values  from  the  two  ratings  of 
the  adjective  scales  but  that  large  differences  appeared  in  the  standard  deviations  of  the  distri¬ 
butions  of  mean  ratings.  It  will  be  noted  that  the  standard  deviations  associated  with  the  bench¬ 
mark  scales  are  greater  than  for  the  other  two  rating  occasions  for  each  factor,  in  some  cases 
twice  as  large.  This  confirms  that  the  intent  to  flatten  the  distributions  of  ratings  has  been 
achieved. 

The  Predictive  Efficiency  of  the  Benchmark  Scales 

The  zero-order  correlations  of  the  benchmark  scale  factor  ratings  with  the  criterion  board 
mean  rating  were  significantly  higher  for  almost  all  the  factors  than  the  correlations  obtained 
with  the  other  sets  of  adjective  ratings.  The  correlation  of  the  mean  grade  rating,  obtained  at 
the  time  of  the  benchmark  scale  rating,  with  the  criterion  board  mean  was  slightly  lower  than  in 
the  projection  sample  administration  and  significantly  lower  than  in  the  original  validation  sample. 
These  data  are  reported  in  Table  3. 

The  multiple  validity  of  the  background  variables  selected  for  use  in  the  Officer  Grade 
Requirements  study,  the  supervisor’s  judgment  of  the  appropriate  grade  for  the  job,  and  the  ten 
factors  as  rated  in  each  of  the  three  situations  was  derived  to  assess  the  gain  or  loss  in  pre¬ 
dictive  efficiency  from  the  shift  to  the  benchmark  scaling  technique.  Although  there  is  a  statis¬ 
tically  significant  loss  from  the  value  obtained  in  the  equation  development  sample  (Time  1), 
there  is  no  significant  difference  between  the  benchmark  scale  validity  and  that  derived  from 
the  adjective  scale  at  Time  2.  These  data,  indicating  which  of  the  variables  received  zero 
and  non-zero  regression  weights,  and  the  multiple  correlations  appear  in  Table  4.  The  complete 
list  of  variables,  their  description,  and  their  intercorrelations  appear  in  Appendix  II. 

Because  there  had  been  policy  differences  between  aircrew  and  non-aircrew  raters  in  the 
ordering  of  the  job  title  scales  for  certain  factors,  analyses  were  run  to  determine  if  systematic 
differences  in  the  evaluation  of  aircrew  positions  and  non-aircrew  jobs  occurred.  In  the  OGR 
study  dramatic  changes  in  grade  distributions  in  the  scientific  and  development  engineering 
utilization  fields  were  recomemended.  For  this  reason  both  this  area  and  the  aircrew  area  were 
treated  to  seek  systematic  differences  in  predictive  efficiency.  It  was  found  that  slight  changes 
in  the  level  of  predictive  efficiency  occurred  when  categorical  variables  relating  to  membership 
in  the  scientific  and  engineering  career  fields  were  studied  but  that  the  aircrew  variable  carried 
no  effect.  These  data  appear  in  Table  5. 
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Table  2.  Correlations  Between,  Means,  and  Standard  Deviations  of  Ten 
Job  Requirement  Rating  Factors  and  Field  Grade  Rating 'Across 
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Table  3.  Correlations  Between  Mean  Factor  and  Grade  Ratings 
and  OGR  Criterion  Board  Action 


(  N  =  996) 


OGR 

OGR 

Benchmark 

Projection 

Val  idation 

Factor 

Scale 

Sample 

Sample 

1  Formal  Education 

.61 

.46*  * 

.47** 

2  Special  Training  &  Work  Experience 

.61 

.43** 

.44** 

3  Working  Conditions 

-.03 

-.08 

-.05 

4  Originality,  Ingenuity,  &  Creativeness 

.67 

.52** 

.56** 

5  Communication  Skills 

.72 

.61** 

•64*  * 

6  Interpersonal  Skills 

.69 

.56** 

.59** 

7  Judgment  &  Decision  Making 

.75 

.53** 

.59** 

8  Planning 

.76 

.63** 

.73 

9  Management 

.79 

'Jl*  * 

.78 

10  Risk 

-.22 

-.27 

-.27 

11  Mean  Grade  Rating 

.86 

bo 

CO 

* 

.90** 

**  Value  different  from  that  obtained  for  Benchmark  Scale  at  .01  level  of  confidence. 
’Value  different  from  that  obtained  for  Benchmark  Scale  at  .05  level  of  confidence. 


For  convenience  in  possible  hand  computation,  if  applied  as  a  field  procedure,  it  was 
decided  that  final  grade  determination  should  be  based  upon  composite  scores  composed  of 
the  factor  ratings,  the  background  or  assignment  variables,  the  supervisory  judgment  of  appro¬ 
priate  grade,  and  the  mean  grade  rating  as  assigned  in  the  field.  Only  those  factors  selected 
for  the  projection  equation  in  the  OGR  study  were  included.  In  treating  the  three  rating  sam¬ 
ples  separately,  it  was  found  that  the  level  of  predictive  efficiency  did  not  deteriorate  from 
that  possible  with  the  full  set.  These  data  are  in  Table  6. 

Discovery  that  the  predictive  efficiency  of  the  benchmark  scales  was  not  significantly 
different  from  that  of  the  OGR  equation  used  for  the  projection  values  supported  the  decision 
to  develop  a  system  suitable  for  hand  application  by  manpower  officials  throughout  the  Air 
Force.  To  facilitate  hand  application,  the  variables  reported  in  Table  6  were  treated  to  pro¬ 
duce  integer  weights.  Weights  to  be  used  were  determined  by  dividing  the  regression  weights 
in  each  subcomposite  by  the  smallest  regression  weight  derived  for  that  subcomposite.  To 
further  simplify  hand  processing,  the  organizational  level  variables  and  supervisor’s  judg¬ 
ment  of  appropriate  grade  were  set  up  in  a  two-step  process,  with  the  proper  weight  first  applied 
to  the  data  from  the  background  data  and  history  sheet,  and  the  resultant  data  then  weighted 
into  the  final  equation.  The  integer  weights  for  organizational  level  and  supervisory  judgment 
subcomposites  appear  in  Table  7. 

The  values  derived  from  these  composites  are  combined  with  the  evaluations  of  the  fac¬ 
tors  and  grade  ratings  as  obtained  from  the  field  by  use  of  the  integer  weights  reported  in 
Table  8.  The  predictive  efficiency  of  the  complete  equation  was  found  to  be  .90  when  integer 
weights  were  applied  as  shown  in  this  table. 
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Table  4 ■  Comparison  of  Predictive  Efficiency  of  Rating  Factors  and 
Background  Variables  for  OGR  Grades 

(N  =  996) 


Variable  Weighted3 

Variable 

Time  3b 

Time  2 

Time  1 

Formal  Education 

0 

X 

0 

Special  Training  &  Work  Experience 

0 

0 

0 

Working  Conditions 

X 

0 

X 

Originality,  Ingenuity,  &  Creativeness 

X 

X 

X 

Communication  Skills 

0 

0 

X 

Interpersonal  Skills 

0 

X 

0 

Judgment  &  Decision  Making 

X 

X 

0 

Planning 

0 

X 

X 

Management 

X 

X 

X 

Risk 

0 

X 

0 

Field  Grade  Rating 

X 

X 

X 

Level  of  Organization  in  Air  Force 

X 

X 

X 

Level  of  Job  in  Organization 

X 

X 

X 

Supervisors  Judged  Grade 

X 

X 

X 

Supervisors  Judged  Grade  Lieutenant 

X 

X 

X 

Supervisors  Judged  Grade  Captain 

X 

X 

X 

Supervisors  Judged  Grade  Major 

X 

0 

0 

Supervisors  Judged  Grade  Lieutenant  Colonel 

X 

X 

X 

Supervisors  Judged  Grade  Colonel 

X 

X 

X 

Supervisors  Judged  Grade  General 

0 

0 

0 

R2 

.82 

.83 

.86 

R 

.90 

.91 

•93 

a  Regression  weights  not  given;  "0”  means  Variable  received  zero  weight  in  computation,  "X”  means 
a  non-zero  weight. 

bTime  3  =  Benchmark  Scales,  ratings  collected  Oct  65  through  Feb  66. 

Time  2  =  OGR  Projection  Sample,  adjective  scale,  ratings  collected  Oct  through  Dec  64. 

Time  1  =  OGR  Equation  Development  Sample,  adjective  scale,  ratings  collected  Jan  through 
Mar  64. 
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Table  5.  Contribution  of  Membership  Variables  in  Aircrew  and  Scientific  and 
Development  Engineering  Career  Fields  to  Efficiency  of  Grade  Prediction 


Variable 

— 

Variobl 

Benchmark 

e  Weighted3 

Rated  Data  Only 

Formal  Education 

0b 

oc 

0d 

oe 

0f 

og 

0h 

Special  Training  &  Work  Experience 

0 

0 

0 

0 

0 

0 

0 

Working  Conditions 

X 

X 

X 

X 

X 

X 

X 

Originality,  Ingenuity,  &  Creativeness 

X 

X 

X 

X 

X 

X 

X 

Communication  Skills 

0 

0 

0 

0 

0 

0 

0 

Interpersonal  Skills 

0 

0 

0 

0 

0 

X 

0 

Judgment  &  Decision  Making 

X 

X 

X 

X 

X 

X 

X 

Planning 

0 

0 

0 

0 

0 

0 

0 

Management 

X 

X 

X 

X 

X 

X 

X 

Risk 

0 

0 

0 

0 

0 

X 

0 

Mean  Grade  Rating 

X 

X 

X 

X 

X 

X 

X 

Level  of  Organization  in  Air  Force 

X 

X 

X 

X 

X 

X 

X 

Level  of  Job  in  Organization 

X 

X 

X 

X 

X 

X 

X 

Supervisors  Judged  Grade 

X 

X 

X 

X 

X 

X 

X 

Supervisors  Judged  Grade  Lieutenant 

X 

X 

X 

X 

X 

X 

X 

Supervisors  Judged  Grade  Captain 

X 

X 

X 

X 

X 

X 

X 

Supervisors  Judged  Grade  Major 

X 

X 

X 

X 

X 

X 

X 

Supervisors  Judged  Grade  Lieutenant  Colonel 

X 

X 

X 

X 

X 

X 

X 

Supervisors  Judged  Grade  Colonel 

X 

X 

X 

X 

X 

X 

X 

Supervisors  Judged  Grade  General 

0 

0 

0 

0 

0 

0 

0 

Pilot  Jobs1  (AFSC  10XX,  11XX,  12XX,  13XX,  14XX) 

X 

X 

X 

.. 

-- 

-- 

-- 

Navigator  Jobs1  (AFSC  15XX) 

X 

X 

-- 

X 

-- 

-- 

-- 

Pilot  and  Navigator  Jobs  1 

-- 

-- 

-- 

-- 

-- 

X 

-- 

Scientific  &  Development  Engineering  Jobs1 
(AFSC  26XX,  27XX,  28XX) 

X 

X 

R2 

.82 

.82 

.82 

.82 

.82 

.82 

.82 

R 

.90 

.90 

.90 

.90 

.90 

.90 

.90 

aRegression  weights  not  given;  "0”  means  variable  received  zero  weight  in  computation,  "X”  means 
a  non-zero  weight,  means  variable  was  not  available  in  the  predictor  set  for  the  problem  presented. 

b  Pilot,  navigator,  and  S&DE  categories  included. 

c  Pilot  and  navigator  categories  included. 

d  Pilot  category  included. 

e  Navigator  category  included. 

1  Scientific  &  Development  Engineering  category  included. 
g  Combined  pilot  and  navigator  categories  included. 
h Included  no  data  on  membership  in  special  categories. 

1  Categorical  Membership  variable,  1  if  member,  0  if  not. 
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Table  6.  Predictive  Efficiency  in  the  Three  Rating  Samples  of  the  Variables 
Included  in  the  Final  Equation  of  the  OGR  Study 

(N  =  996  Jobs) 


Variabl  e 

Time  3  a 

Time  2 

Time  1 

Special  Training  &  Work  Experience 

0 

0 

0 

Communication  Skills 

X 

X 

0 

Judgment  &  Decision  Making 

X 

X 

0 

Planning 

X 

X 

X 

Management 

X 

X 

X 

Mean  Grade  Rating 

X 

X 

X 

Level  of  Organization  in  Air  Force 

X 

X 

0 

Level  of  Job  in  Organization 

X 

X 

X 

Supervisors  Judged  Grade  Lieutenant 

X 

X 

X 

Supervisors  Judged  Grade  Captain 

X 

X 

X 

Supervisors  Judged  Grade  Major 

X 

X 

0 

Supervisors  Judged  Grade  Lieutenant  Colonel 

X 

X 

X 

Supervisors  Judged  Grade  Colonel 

X 

X 

X 

Supervisors  Judged  Grade  General 

0 

0 

0 

R2 

.82 

.83 

.86 

R 

.90 

.91 

.93 

fl  Time  3  =  Benchmark  Scales,  ratings  collected  Oct  65  through  Feb  66. 

Time  2  =  OGR  Projection  Sample,  adjective  scale,  ratings  collected  Oct  through  Dec  64. 

Time  1  =  OGR  Equation  Development  Sample,  adjective  scale,  ratings  collected  Jan  through  Mar  64. 


Table  7.  Integer  Weights  for  Subcomposites  of  Organizational  Level 
and  Supervisor’s  Judged  Grade  for  Position 


Variable 

Integer  Weight 

Organizational  Levels  Subcomposite 

Level  of  Organization  in  Air  Force 

i 

Level  of  Job  in  Organization 

i 

Supervisor’s  Judgment  Subcomposite 

Supervisor’s  Judged  Grade  Lieutenant 

-3 

Supervisor’s  Judged  Grade  Captain 

-2 

Supervisor’s  Judged  Grade  Major 

1 

Supervisor’s  Judged  Grade  Lieutenant  Colonel 

5 

Supervisor’s  Judged  Grade  Colonel 

10 
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Table  8.  Integer  Weights  for  Final  Grade  Prediction  Equation 


Variable  Integer  Weight 


Rated  Data  from  Field 

Special  Training  and  Work  Experience  3 

Communication  Skills  4 

Judgment  and  Decision  Making  2 

Planning  1 

Management  12 

Mean  Grade  Rating  from  Field  4 

Organizational  Levels  Subcomposite0  1 

Supervisor’s  Judgment  Subcomposite  4 


0  Computed  according  to  weights  given  in  Table  7. 

Note. -Correlation  of  this  composite,  computed  with  integer  weights  for  996  jobs 
of  this  study,  with  the  mean  OGR  grade  =  .90. 


III.  SUMMARY 


A  set  of  job  requirement  factor  scales  based  upon  a  benchmark  scale  presenting  job  titles 
to  identify  successive  levels  of  requirement  were  derived.  These  scales  were  applied  to  1000 
Air  Force  officer  position  descriptions  collected  and  previously  applied  in  the  Officer  Grade 
Requirements  study.  Comparison  of  rating  distributions  based  upon  adjective  scales  and  the 
benchmark  scales  revealed  lower  mean  values,  larger  standard  deviations,  and  superior  zero- 
order  validity  of  the  ratings  based  on  the  benchmark  scales.  The  predictive  efficiency  of  optimal 
composites  of  the  benchmark  scales  for  both  full  sets  of  factors  and  the  subset  chosen  for  appli¬ 
cation  in  the  Officer  Grade  Requirements  study  was  equivalent  to  that  of  the  adjective  scales.  A 
set  of  integer  weights  for  use  in  field  application  of  the  equation  was  derived  without  significant 
loss  of  validity. 
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APPENDIX  I:  BENCHMARK  SCALES  AND  INSTRUCTIONS 


JOB  REQUIREMENT  FACTORS 
Benchmark  Scales 


DIRECTIONS 

1.  Find  the  sheet  in  your  packet  which  is  headed  ASSIGNMENT 
INFORMATION.  Complete  the  first  portion  of  this  form  by  filling 
in  all  the  blocks  except  REMARKS,  FACTOR  RATINGS,  and  GRADE  CODE 

by  checking  appropriate  boxes  and  neatly  printing  requested  information. 

2.  Carefully  read  the  first  job  description  in  your  packet  and  rate 
it  on  job  requirement  FACTOR  1.  FORMAL  EDUCATION.  Notice  that  each 
of  the  9  levels  on  the  factor  are  defined  by  titles  of  benchmark  jobs. 

A  rating  of  "V  on  Factor  1  would  imply  that  you  feel  a  job  requires 
less  formal  education  than  benchmark  jobs  listed  in  levels  5  through 
9.  It  would  require  more  formal  education  than  jobs  listed  at  levels 
1  through  3,  and  about  the  same  amount  of  formal  education  as  jobs 
listed  at  level  4.  The  benchmark  jobs  should  be  weighed  carefully  in 
making  your  decision.  Jobs  which  are  cited  involving  obsolete  equip¬ 
ment  should  be  considered  as  they  were  before  such  units  became  obsolete. 

3.  Next,  rate  the  first  job  on  Factors  2  through  10.  In  each  instance 
study  the  factor  definition  and  the  benchmark  jobs  before  making  your 
rating. 

4.  Now  decide  what  grade  of  Air  Force  officer  should  be  assigned  to 
the  job  described.  Do  this  by  writing  the  proper  number  from  the  Grade 
Code  Listing  in  the  GRADE  CODE  box  for  each  job.  Notice  in  the  Grade 
Code  that  there  are  three  numbers  associated  with  each  grade  level 
except  General.  The  three  numbers  in  a  set  are  used  to  show  three 
levels  of  experience.  For  instance,  a  4_  represents  a  captain  with  a 
short  time  in  grade,  a  5.  represents  a  captain  with  an  average  time  in 
grade,  and  a  6_ represents  a  captain  with  a  long  time  in  grade. 

5.  Now  rate  the  remaining  four  jobs  in  your  packet  on  the  ten  job 
requirement  factors  and  on  the  grade  rating  code.  When  you  have 
completed  the  ratings,  answer  the  questions  in  the  REMARKS  section 

of  the  form.  Gather  all  materials  into  the  inclosed  envelope  and  return 
to  the  Personnel  Research  Laboratory. 


This  Study  Authorized  under  AFR  0-7,  1  Apr  1965 
AFPT  80-04l(A),  1  Jan  1966 


II 


FACTOR  1:  FORMAL  EDUCATION:  The  amount  of  formal  education  required  by  the  job.  Consider 
the  education  obtained  in  high  school,  college,  university,  or  professional  school. 

LEVEL  9 

Chief,  Computer  Techniques  Div,  Hq  Air  Weather  Service 
Industrial  Engineer,  Hq  Air  Base  Wg 

Chief,  Re-entry  Vehicle  Div,  Strategic  Missile  Evaluation  Sq 

LEVEL  8 

Nuclear  Research  Officer,  Research  Technology  Div,  AF  Flight  Dynamics  Lab 
Chief,  Military  Affairs  &  Asst  Staff  Judge  Advocate,  Air  Base  Gp 
Chief  Physiological  Chemistry  Dept,  USAF  School  Aerospace  Medicine 

LEVEL  7 

Deputy  Commander  for  Maintenance,  Strategic  Wg 

Electronic  Systems  Installation  Officer,  Ground  Electronics  Engineering  Installation  Agency  Sq 
Chief,  Munitions  Div,  Hq  Numbered  Air  Force 

LEVEL  6 

Chief  Data  Systems  &  Statistics,  Combat  Support  Gp 

Base  Vice  Commander,  Combat  Support  Gp 

Asst  Staff  Electronics  Officer,  Combat  Evaluation  Gp 

LEVEL  5 

Assistant  DCS/Personnel,  Hq  Air  Weather  Service 

Aerial  Reconnaissance  Weather  Officer,  Weather  Reconnaissance  Sq 

Armament  Staff  Officer,  Inspector  General  Gp 

LEVEL  4 

OIC  Armament  &  Electronics  Branch,  Consolidated  Aircraft  Maintenance  Sq 
Director  of  Personnel,  Hq  Combat  Support  Gp 
Electronic  Warfare  Officer,  Strategic  Reconnaissance  Sq 

LEVEL  3 

Chief,  Photographic  Services  Branch,  Aerospace  Reconnaissance  Technical  Wg 
Chief  Munitions  Maintenance  Branch,  Munition  Maintenance  Sq 
Organizational  Maint  Officer,  Interceptor  Fighter  Sq 

LEVEL  2 

Chief  Transportation  Traffic  Management,  Transportation  Sq 
Group  Supply  Officer,  Aeromedical  Evacuation  Gp 
Base  Fuels  Officer,  Fighter  Wg 

LEVEL  1 

Automotive  Maintenance  Officer,  Transportation  Sq 
Food  Service  Officer,  Combat  Support  ^Gp 
Clothing  Sales  Officer,  Combat  Support  Gp 
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FACTOR  2:  SPECIAL  TRAINING  AND  WORK  EXPERIENCE:  The  extent  to  which  the  job  requires 
knowledges  and  skills  which  must  be  acquired  through  special  training  courses  or  on-the-job  experi¬ 
ence.  Disregard  general  courses  given  by  Squadron  Officer  School,  Command  and  Staff  College,  or 
War  College. 


LEVEL  9 

Chief,  Contract  Pricing  Branch,  Hq  USAF 
Chief,  Military  Justice  Division,  Air  Div 

Space  Vehicle  Research  Officer,  Hq  AF  Special  Weapons  Center 

LEVEL  8 

Chief,  Geodetic  Survey  Div,  Geodetic  Survey  Sq 

Minuteman  Trajectory  Engineer,  Aerospace  Tech  Wg 

Chief,  Missile/Nuclear  Safety  Division,  Technical  Training  Center 

LEVEL  7 

Missile  Combat  Crew  Commander,  Support  Sq 

Chief,  Target  Intelligence  Branch,  Strategic  Aerospace  Wg 

Pilot,  B-58  Bomber,  Bomb  Sq 

LEVEL  6 

Wing  Director  of  Personnel,  Bomb  Wg 

Flying  Safety  Officer,  Hq  Tactical  Fighter  Wg 

Reconnaissance  Aircraft  Commander,  Support  Sq 

LEVEL  5 

Co-pilot  B-52,  Bomb  Sq 

Pilot,  Troop  Carrier,  Troop  Carrier  Sq 

Radar  Evaluation  Officer,  Hq  Major  Air  Command  (Overseas) 

LEVEL  4 

Co-pilot  KC-97,  Air  Refueling  Sq 
Electronic  Warfare  Officer  B-52,  Bomb  Sq 
Radar  Intercept  Officer,  Fighter  Intercept  Sq 

LEVEL  3 

Crypto  Operations  Officer,  Communications  Gp  (Overseas) 

Avionics  Officer,  Consolidated  Aircraft  Maintenance  Sq 
Flight  Line  Maintenance  Officer,  Organizational  Maintenance  Sq 

LEVEL  2 

Photographic  Equipment  Maintenance  Officer,  Armament  &  Electronic  Maintenance  Sq 
Chief,  Pay  &  Travel  Branch,  Combat  Support  Gp 
Photographic  Officer,  Technical  Reconnaissance  Sq 

LEVEL  1 

Base  Housing  Officer,  Combat  Support  Gp 
Personnel  Services  Officer,  Fighter  Gp 
Transportation  Officer,  Instrumentation  Sq 
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FACTOR  3:  WORKING  CONDITIONS:  The  extent  to  which  the  job  involves  uncomfortable  working 
conditions.  Consider  such  conditions  as  isolation,  irregular  hours,  monotony,  prolonged  vigilance, 
extensive  TDY,  and  pressure  to  meet  deadlines. 


LEVEL  9 

B-58  Aircraft  Commander,  Bomb  Sq 
Co-pilot  KC-135,  Air  Refueling  Sq 
Commanding  Officer,  B-52  Bomb  Sq 

LEVEL  8 

Pilot,  Reconnaissance,  Tactical  Reconnaissance  Sq  (Overseas) 
Tactical  Fighter  Pilot,  Tactical  Fighter  Sq 
B-26  Instructor  Pilot,  Fighter  Sq  (Commando) 

LEVEL  7 

Pilot,  Search  and  Rescue,  Air  Rescue  Sq 

Weapons  Controller,  Air  Early  Warning  and  Control  Sq 

Transport  Instructor  Navigator,  Troop  Carrier  Sq 

LEVEL  6 

Electrical  Engineer,  Site  Activation  Task  Force 
Instructor  Pilot  C-97  Aircraft,  Operations  Sq 
Chief,  Division  Intelligence,  Hq  Air  Div  (Overseas) 

LEVEL  5 

Weather  Forecaster,  Det,  Weather  Gp 
Chief,  Logistics  Division,  Aerospace  Wg 
Nurse  Anesthetist,  Medical  Group 

LEVEL  4 

Transportation  Officer,  Transportation  Sq 

Director  of  Manpower  and  Organization,  Hq  Air  Div  SAGE 

Chief  Airman  Personnel  Division,  Hq  CONAC 

LEVEL  3 

Asst  Staff  Judge  Advocate,  Combat  Support  Gp 

Clinical  Psychologist,  Aerospace  Medical  Gp 

Chief  Military  Justice  Division,  Hq  AF  Missile  Test  Center 

LEVEL  2 

Personnel  Services  Officer,  Air  Base  Gp 
Pharmacy  Officer,  Numbered  USAF  Hospital 
Staff  Chaplain,  Hq  EASTAF 

LEVEL  1 

Officers’  Open  Mess  Secretary,  Support  Gp 
Custodian,  Non  Appropriated  Funds,  Air  Base  Wg 
Information  Officer,  Bomb  Wg 
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FACTOR  4:  ORIGINALITY,  INGENUITY,  AND  CREATIVENESS:  The  extent  to  which  the  job 
requires  new  and  unique  methods,  approaches,  and  solutions  to  problems.  Consider  the  demand 
for  novel  ideas  and  inventiveness. 


LEVEL  9 

Research  Aviation  Physiologist,  USAF  School  of  Aviation  Medicine 
Astronautical  Engineer,  Propulsion,  Hq  AF  Missile  Development  Center 
Human  Performance  Engineer,  Electronic  Systems  Div 

LEVEL  8 

Logistic  Staff  Officer,  Hq  Air  Materiel  Area 

Manpower  Management  Staff  Officer,  Hq  Major  Air  Command 

Director,  Department  of  Aircraft  Maintenance  Training,  Tech  School 

LEVEL  7 

Missile  Safety  Officer,  Hq  Air  Proving  Ground  Center  Safety  Div 
Base  Deputy  Commander  for  Materiel,  Combat  Support  Gp 
Management  Engineering  Officer,  Hq  Air  Materiel  Area 

LEVEL  6 

Civil  Engineer,  Civil  Engineer  Sq 

Chief  Re-Entry  Vehicle  Maintenance  Branch,  Missile  Maintenance  Sq 
Flying  Safety  Officer,  Hq  Tactical  Fighter  Wg 

LEVEL  5 

Missile  Combat  Crew  Commander,  Strategic  Missile  Sq 
Reconnaissance  Pilot,  Day  Photo  Jet,  Tactical  Reconnaissance  Sq 
Commander,  Weather  Sq 

LEVEL  4 

Precision  Photographic  Services  Officer,  Hq  Sq,  Strategic  Wg 
Chief  Transportation  Traffic  Management,  Transportation  Sq 
Computer  Maintenance  Officer,  Support  Sq 

LEVEL  3 

Launch  Area  Maintenance  Officer,  Air  Defense  Missile  Sq 
Academic  Instructor-Undergraduate  Pilot  Training,  Student  Sq 
Clinical  Laboratory  Officer,  Numbered  USAF  Hospital 

LEVEL  2 

Asst  Base  Equipment  Management  Officer,  Supply  Sq 
Co-Pilot  B-52  Heavy  Bomber,  Bomb  Sq 
Accounting  &  Finance  Officer,  Combat  Support  Gp 

LEVEL  1 

Asst  Medical  Supply  Officer,  Medical  Gp 
Optometry  Officer,  Numbered  USAF  Hospital 
Administrative  Officer,  Air  Base  Gp 
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FACTOR  5:  COMMUNICATION  SKILLS:  The  extent  to  which  the  job  requires  skill  in  oral  and 
written  communication.  Consider  the  complexity  and  variety  of  information  communicated  as  well 
as  the  level  of  the  individuals  and  agencies  involved. 


LEVEL  9 

Command  Director  of  Information,  Hq  Major  Air  Command 
Political  Military  Affairs  Officer,  Hq  USAF 
Secretary  of  the  Air  Staff,  Hq  USAF 

LEVEL  8 

Chief  of  Logistics  Division,  Hq  Numbered  Air  Force 
Astronautical  Engineer,  Hq  Research  Technology  Div 
OSI  District  Commander,  Hq  District  OSI 

LEVEL  7 

Base  Civil  Engineer,  Support  Gp 

Human  Performance  Engineer,  Electronic  Systems  Div 
Wing  Comptroller,  Air  Base  Wg 

LEVEL  6 

Aviation  Physiologist,  Inspector  General  Gp 
Chemical  Engineer,  AF  Aero  Propulsion  Lab 

Administrative  Services  Officer,  Ground  Electronics  Engineering  Installation  Agency 

LEVEL  5 

Officer  Selection  Officer,  Det,  USAF  Recruiting  Gp 

Deputy  Commander,  Strategic  Missile  Sq 

Chief,  Sensors  Section,  AF  Special  Weapons  Center 

LEVEL  4 

Construction  Engineer,  Civil  Engineering  Sq  (Overseas) 

Squadron  Operations  Officer,  Combat  Crew  Training  Sq 
Accounting  &  Finance  Officer,  Fighter  Gp 

LEVEL  3 

Commercial  Transportation  Officer,  Material  Sq 
Avionics  Officer,  Air  Early  Warning  Control  Wg 
Missile  Maintenance  Control  Officer,  Strategic  Missile  Sq 

LEVEL  2 

Fighter  Interceptor  Pilot,  Fighter  Interceptor  Sq 

Strategic  Missile  Complex  Maintenance  Officer,  Strategic  Missile  Sq 
Electronic  Warfare  Officer,  Tactical  Reconnaissance  Sq 

LEVEL  1 

Co-pilot,  Air  Refueling  Sq 
Navigator,  Bomb  Sq 

Helicopter  Pilot  Flight  Commander,  Logistic  Supply  Gp 
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FACTOR  6:  INTERPERSONAL  SKILLS:  The  extent  to  which  the  job  requires  skill  in  dealing 
with  people.  Consider  the  need  for  sensitiveness,  responsiveness,  persuasiveness,  self-control, 
and  tact,  as  well  as  the  possible  consequences  when  such  skills  are  not  employed. 


LEVEL  9 

Staff  Chaplain,  Hq  WESTAF 

Political  Military  Affairs  Officer,  Hq  USAF 

Wing  Commander,  Air  Refueling  Wg 

LEVEL  8 

Asst  Professor  of  Economics,  USAFA 

Academic  Instructor,  Dept  of  Chemistry  &  Physiology,  USAFA 
Security  Staff  Officer,  Hq  Major  Air  Command 

LEVEL  7 

Commissary  Officer,  Air  Base  Gp 
Wing  Director  of  Safety,  Bomb  Wg 
OSI  Detachment  Commander,  OSI  Det 

LEVEL  6 

Base  Procurement  Officer,  Navigator  Training  Wg 
Instructor  Navigator  Bombardier,  Navigator  Training  Wg 
Comptroller,  Air  Materiel  Area 

LEVEL  5 

Personnel  Officer,  Radar  Bomb  Scoring  Sq 

Base  Supply  Officer,  Combat  Support  Gp 

Chief  Accounting  and  Finance  Division,  Combat  Support  Gp 

LEVEL  4 

Personnel  Services  Officer,  Support  Sq 
Chief,  Medical  Materiel  Services,  Medical  Gp 

Maintenance  Supervisor,  Armament  and  Electronics  Maintenance  Sq 

LEVEL  3 

Air  Freight  Supervisor,  Air  Terminal  Sq 

Airborne  Electronics  Maintenance  Officer,  Fighter  Interceptor  Sq 
Construction  Engineer,  Civil  Engineering  Sq  (Overseas) 

LEVEL  2 

OIC  Photo  Laboratory  Section,  Reconnaissance  Tech  Sq 
Avionics  Officer,  Armament  &  Electronics  Maint  Sq 
Precision  Photographic  Services  Officer,  Strategic  Wg 

LEVEL  1 

OIC  Weapons  Services  Branch,  Munition  Maintenance  Sq 
Navigator,  Air  Refueling  Sq 
Co-pilot,  Air  Refueling  Sq 
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FACTOR  7:  JUDGMENT  AND  DECISION  MAKING:  The  importance  and  independence  of  judgments 
and  decisions  required  by  the  job.  Consider  the  nature,  variety,  and  possible  impact  of  decisions. 
The  less  well  defined  the  guidance  for  decisions,  the  higher  should  be  the  rating;  while  the  more 
specific  and  detailed  the  guidance,  the  lower  should  be  the  rating. 


LEVEL  9 

Chief,  Budget  Div,  Hq  Major  Air  Command 

Staff  Legal  Officer,  Military  Affairs,  Hq  USAF 

Chief,  Weapon  System  Testing  Div,  Ballistic  Systems  Div 

LEVEL  8 

Base  Vice  Commander,  Combat  Support  Gp 

Missile  Maintenance  Inspector,  IG,  Hq  Major  Air  Command 

DCS/Comptroller,  Hq  Numbered  Air  Force  (Overseas) 

LEVEL  7 

Logistics  Officer,  Hq  Ballistic  Systems  Div 

Experimental  Flight  Test  Officer,  Hq  Aerospace  Systems  Div 

Director  of  Personnel,  Combat  Support  Gp 

LEVEL  6 

Commander,  Organizational  Maintenance  Sq 
Missile  Safety  Officer,  Strategic  Missile  Wg 
Missile  Combat  Crew  Commander  (ICBM),  Strategic  Missile  Sq 

LEVEL  5 

Site  Maintenance  Officer,  Strategic  Missile  Wg 
Base  Operations  Officer,  Combat  Support  Gp 
Aircraft  Commander  KC-135,  Air  Refueling  Sq 

LEVEL  4 

Munitions  Maintenance  Supervisor,  Munitions  Maintenance  Sq 
Aircraft  Commander  U-2,  Strat  Recon  Weather  Sq 
Fighter  Interceptor  Pilot,  Fighter  Interceptor  Sq 

LEVEL  3 

Reconnaissance  Pilot,  Mapping  &  Charting  Sq 
Pilot,  Transport,  Troop  Carrier  Sq 
Chief,  Base  Services  Div,  Support  Sq 

LEVEL  2 

Traffic  Management  Officer,  Transportation  Sq 

Navigator,  Bomb  Sq 

Helicopter  Pilot,  Combat  Support  Gp 

LEVEL  1 

Recreation  Services  Officer,  Combat  Support  Gp 
Pharmacy  Officer,  USAF  Dispensary 
Photographic  Officer,  Technical  Reconnaissance  Sq 
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FACTOR  8:  PLANNING:  The  extent  to  which  planning  is  required  by  the  job.  Consider  the  scope 
and  significance  of  work  for  which  planning  is  done.  The  longer  the  time  span  for  which  planning  is 
done,  the  higher  the  rating  should  be. 


LEVEL  9 

Deputy  Chief,  Plans  Division,  Hq  Major  Air  Command 

Asst  Director  of  War  Plans,  Hq  Major  Air  Command 

Director  Gemini  Launch  Vehicle  Directorate,  Hq  Space  Systems  Div 

LEVEL  8 

Chief,  R  &  D  Contracts  Div,  Hq  AF  Special  Weapons  Center 
Management  Engineering  Officer,  Hq  Air  Materiel  Area 
Wing  Logistics  Officer,  Air  Refueling  Wg 

LEVEL  7 

Maintenance  Control  Officer,  Missile  Training  Wg 
Base  Vice  Commander,  Combat  Support  Gp 
Budget  Officer,  Air  Base  Gp 

LEVEL  6 

Operations  Officer,  Fighter  Interceptor  Sq 
Hospital  Administrator,  Numbered  USAF  Hospital 
Chief,  Data  Services  Division,  Combat  Support  Gp 

LEVEL  5 

Chief,  Officer  Personnel  Branch,  Air  Base  Gp 
Traffic  Management  Officer,  Transportation  Sq 
Procurement  Officer,  Combat  Support  Gp 

LEVEL  4 

Missile  Combat  Crew  Commander,  Strategic  Missile  Sq 
Wing  Administration  Officer,  Troop  Carrier  Wg 
Aircraft  Commander,  Defense  Sys  Eval  Sq 

LEVEL  3 

Electronic  Warfare  Officer,  Bomb  Sq 

Medical  Administrative  Officer,  USAF  Dispensary 

Reconnaissance  Pilot,  Day  Jet,  Tactical  Recon  Sq 

LEVEL  2 

Fighter  Interceptor  Pilot,  Fighter  Interceptor  Sq 

Registrar,  Medical  Gp 

Air  Police  Officer,  Combat  Defense  Sq 

LEVEL  1 

Asst  Photo  Officer,  Armament  &  Electronic  Maintenance  Sq 
Weather  Forecaster,  Det,  Weather  Sq 
Optometrist,  Medical  Gp 
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FACTOR  9:  MANAGEMENT:  The  level  of  executive,  and  managerial  skills  required  in  the  job. 
Consider  the  complexity,  variety,  and  level  of  the  activities  which  are  directed,  organized, 
coordinated,  controlled,  commanded,  or  evaluated. 


LEVEL  9 

Director  of  Budget,  Hq  Major  Air  Command 
Commander,  Combat  Support  Gp  (Overseas) 

Wing  Commander,  Strategic  Aerospace  Wg 

LEVEL  8 

Chief,  Manpower  Validation  Team,  Support  Sq 
Chief  of  Operations,  Air  Defense  Missile  Sq 
Group  Executive  Officer,  Air  Base  Gp 

LEVEL  7 

Maintenance  Supervisor,  Armament  &  Electronics  Maintenance  Sq 
Squadron  Operations  Officer,  Combat  Crew  Training  Sq 
Base  Accounting  &  Finance  Officer,  Flying  Training  Wg 

LEVEL  6 

Deputy  Director  of  Personnel,  Combat  Support  Gp 

Base  Procurement  Officer,  Pilot  Training  Wg 

Helicopter  Squadron  Operations  Officer,  Flying  Training  Sq 

LEVEL  5 

Traffic  Management  Officer,  Transportation  Sq 

Base  Communications  Maintenance  Officer,  Communications  Sq  (Overseas) 
Missile  Combat  Crew  Commander,  Strategic  Missile  Sq 

LEVEL  4 

OIC  Utilities  Operations  Division,  Civil  Engineering  Sq 
OIC  Photo  Evaluation  Branch,  Mapping  &  Charting  Sq 
Base  Fuels  Officer,  Supply  Sq 

LEVEL  3 

Primary  Pilot  Training  Instructor,  Pilot  Training  Sq 
Space  Surveillance  Officer,  Aerospace  Control  Sq 
Air  Traffic  Controller,  Communication  Sq 

LEVEL  2 

Administrative  Officer,  Air  Base  Sq 
Data  Services  Officer,  Combat  Support  Gp 
Tactical  Fighter  Pilot,  Tactical  Fighter  Sq 

LEVEL  1 

Clinical  Psychologist,  Named  USAF  Hospital 
Psychiatric  Social  Worker,  Named  USAF  Hospital 
Helicopter  Pilot  Single  Rotor,  Air  Base  Sq 
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FACTOR  10:  RISK:  The  extent  to  which  the  job  requires  exposure  to  risk  of  death  or  severe  inj  ury 
in  peace-time. 


LEVEL  9 

Pilot  KC-135,  Air  Refueling  Sq 

Tactical  Fighter  Pilot,  Tactical  Fighter  Sq 

Instructor  Pilot,  Tactical  Fighter,  Combat  Crew  Training  Sq 

LEVEL  8 

Co-pilot  KC-97,  Air  Refueling  Sq 
Navigator,  Troop  Carrier,  Troop  Carrier  Sq 
Pilot  C-133,  Air  Transport  Service 

LEVEL  7 

Instructor  Missile  Launch  Officer,  Strategic  Missile  Sq 
Arctic  Survival  Training  Officer,  Strategic  Wg  (Overseas) 

Chief,  Propellants  Programming  Br,  AF  Rocket  Propulsion  Laboratory 

LEVEL  6 

Commander,  Radar  Sq 

Research  Biochemist,  School  of  Aerospace  Medicine 
Chemist,  Air  Force  Materials  Laboratory 

LEVEL  5 

Chief,  General  Investigations  Div,  Hq  District  OSI 
Base  Veterinarian,  Numbered  USAF  Hospital 
Commander,  Civil  Engineering  Sq 

LEVEL  4 

Mechanical  Engineer,  Hq  AF  Missile  Development  Center 
Air  Traffic  Control  Officer,  Communications  Sq 
Instructor,  Warfare  Systems  School 

LEVEL  3 

Medical  Supply  Officer,  Numbered  USAF  Hospital 

Chief,  Engineering  Standards  Branch,  Communications  Region 

Chief  Machine  Processing,  Support  Gp 

LEVEL  2 

Recreation  Services  Officer,  Combat  Support  Gp 
Chaplain,  Combat  Support  Gp 

Manpower  Management  Staff  Officer,  Flying  Training  Wg 

LEVEL  1 

Custodian  Non  Appropriated  Funds,  Air  Base  Wg 
Clothing  Sales  Store  Officer,  Air  Base  Wg 
Instructor,  French,  Dept  of  Foreign  Languages,  USAFA 
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APPENDIX  II:  VARIABLES  AND  INTERCORRELATIONS 


Variable  Name 

Mean 

S.D. 

1  Formal  Education  (T3)S 

5.16 

1.63 

2  Special  Training  and  Work  Experience  (T3) 

5.80 

1.42 

3  Working  Conditions  (T3) 

5.21 

1.76 

4  Originality,  Ingenuity,  and  Creativeness  (T3) 

5.14 

1.64 

5  Communication  Skills  (T3) 

5.30 

1.83 

6  Interpersonal  Skills  (T3) 

5.58 

1.71 

7  Judgment  and  Decision  Making  (T3) 

5.41 

1.63 

8  Planning  (T3) 

5.25 

1.72 

9  Management  (T3) 

5.21 

1.86 

10  Risk  (T3) 

4.48 

2.52 

11  Mean  Grade  Rating  (T3) 

7.88 

2.95 

12  Formal  Education  (T2)b 

5.77 

1.04 

13  Special  Training  and  Work  Experience  (T2) 

6.26 

0.80 

14  Working  Conditions  (T2) 

5.16 

1.27 

15  Originality,  Ingenuity,  and  Creativeness  (T2) 

5.43 

1.03 

16  Communication  Skills  (T2) 

5.94 

1.11 

17  Interpersonal  Skills  (T2) 

5.95 

0.99 

18  Judgment  and  Decision  Making  (T2) 

6.00 

0.95 

19  Planning  (T2) 

5.83 

1.08 

20  Management  (T2) 

5.63 

1.20 

21  Risk  (T2) 

4.32 

2.04 

22  Mean  Grade  Rating  (T2) 

7.85 

2.7  6 

23  Formal  Education  (Tl)° 

5.71 

1.01 

24  Special  Training  and  Work  Experience  (Tl) 

6.22 

0.72 

25  Working  Conditions  (Tl) 

5.15 

1.18 

26  Originality,  Ingenuity,  and  Creativeness  (Tl) 

5.37 

0.99 

27  Communications  Skills  (Tl) 

5.81 

1.08 

28  Interpersonal  Skills  (Tl) 

5.89 

0.94 

29  Judgment  and  Decision  Making  (Tl) 

5.87  . 

0.91 

30  Planning  (Tl) 

5.71 

1.03 

31  Management  (Tl) 

5.54 

1.17 

32  Risk  (Tl) 

4.34 

1.96 

33  Mean  Grade  Rating  (Tl) 

7.66 

2.71 

34  Level  of  Organization  in  the  Air  Force  (PV22) 

4.95 

2.08 

35  Level  of  Job  in  Organization  (PV25) 

3.97 

1.72 

36  Supervisors  Judged  Grade  (PV28) 

3.85 

1.17 

37  Supervisors  Judgment  —  Lt  (PV29) 

0.08 

0.26 

38  Supervisors  Judgment  —  Capt  (PV30) 

0.28 

0.45 

39  Supervisors  Judgment  — Maj  (PV31) 

0.34 

0.47 

40  Supervisors  Judgment  — Lt  Col  (PV32) 

0.21 

0.41 

41  Supervisors  Judgment  —  Col  (PV33) 

0.08 

0.26 

42  Supervisors  Judgment  —  Gen  (PV34) 

0. 

0. 

43  UMD  Authorized  Grade  (PV35) 

3.67 

1.18 

44  Officer’s  Present  Grade  (PV42) 

3.47 

1.35 

45  Criterion  Board  Mean  Action  (PV76) 

7.59 

2.74 

46  Rating  Factors  Composite 

7.59 

2.23 

47  Organization  Levels  Composite 

7.59 

1.80 

48  Supervisors  Judgment  Composite 

7.59 

2.10 

49  Pilot  Jobs 

0.24 

0.43 

50  Navigator  Jobs 

0.06 

0.23 

51  Pilot  and  Navigator  Jobs 

0.29 

0.46 

52  Scientific  and  Development  Engineering  Jobs 

0.10 

0.30 

aTime  3  =  Benchmark  Scales,  ratings  collected  Oct  1965  through  Feb  1966. 
b  Time  2  =  OGR  Projection  Sample,  adjective  scale,  ratings  collected  Oct  through  Dec  1964. 

cTime  1  =  OGR  Equation  Development  Sample,  adjective  scale,  ratings  collected  Jan  through 
Mar  1964. 
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Intercorrelations  of  Job  Evaluation  Factors,  Background  Variables,  and  Criteria 

(N=  996  Jobs) 
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